ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

(2) University of Illinois at Urbana-Champaign); Ameet Rahane (1); Amit Paradkar (1); Anca Sailer (1); Bekir O. Turkkan (1); Bhavya Bhavya (1); Carlos Fonseca (1); Chandrasekhar Narayanaswami (1); Daby Sow (1); Debanjana Kar (1)

REVIEW 4 major objections 5 minor 3 cited by

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents ITBench, a benchmark that tests AI agents on real-world IT automation tasks, and shows that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps…

desk verdict A genuinely useful multi-persona IT benchmark with a fixable metric-labeling problem and too little public data to verify the headline numbers. read the letter →

arxiv 2502.05352 v1 pith:3Z2ODSFK submitted 2025-02-07 cs.AI cs.DCcs.MA

Saurabh Jha (1) , Rohan Arora (1) , Yuji Watanabe (1) , Takumi Yanagawa (1) , Yinfang Chen (2) , Jackson Clark (2) , Bhavya Bhavya (1) , Mudit Verma (1)

show 36 more authors

Harshit Kumar (1) Hirokuni Kitahara (1) Noah Zheutlin (1) Saki Takano (1) Divya Pathak (1) Felix George (1) Xinbo Wu (2) Bekir O. Turkkan (1) Gerard Vanloo (1) Michael Nidd (1) Ting Dai (1) Oishik Chatterjee (1) Pranjal Gupta (1) Suranjana Samanta (1) Pooja Aggarwal (1) Rong Lee (1) Pavankumar Murali (1) Jae-wook Ahn (1) Debanjana Kar (1) Ameet Rahane (1) Carlos Fonseca (1) Amit Paradkar (1) Yu Deng (1) Pratibha Moogi (1) Prateeti Mohapatra (1) Naoki Abe (1) Chandrasekhar Narayanaswami (1) Tianyin Xu (2) Lav R. Varshney (2) Ruchi Mahindru (1) Anca Sailer (1) Laura Shwartz (1) Daby Sow (1) Nicholas C. M. Fuller (1) Ruchir Puri (1) ((1) IBM (2) University of Illinois at Urbana-Champaign)

This is my paper · ORCID

classification cs.AIcs.DCcs.MA

keywords AIagentsITautomationbenchmarkingLLMevaluationSiteReliabilityEngineeringcomplianceassessmentFinOpsincidentresolution

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ITBench is an open benchmarking framework that measures whether AI agents can automate real-world IT operations across three personas: Site Reliability Engineering (diagnosing and mitigating production incidents), CISO compliance (translating security-control requirements into executable policy and checking posture), and FinOps (diagnosing and fixing cloud cost anomalies). The paper's central finding is that agents built on state-of-the-art LLMs resolve only 13.8% of 42 SRE scenarios, 25.2% of 50 CISO scenarios, and 0% of 2 FinOps scenarios, with GPT-4o far ahead of open-weight models but still far from reliable. The authors argue that fully autonomous IT automation is not yet achievable and that progress needs a standardized, extensible benchmark with automated partial-credit evaluation. ITBench formalizes each task as a scenario (a tuple of metadata, a live testbed environment, triggering alerts, and a desired outcome) and scores agents on both exact success and topology-aware partial credit.

What carries the argument

The carrying mechanism is the scenario specification, formalized as a tuple $\langle M, E, T, D\rangle$: $M$ is the metadata and ground truth, $E$ is an operational testbed (a Kubernetes cluster running a demo application with Grafana, Prometheus, Loki, and Jaeger observability), $T$ is a set of golden-signal triggering alerts, and $D$ is the automation goal. The agent interacts with $E$ through natural-language tools (NL2Kubectl, NL2Traces, NL2Metrics, NL2Logs, NL2Alerts), and the interaction is modelled as a POMDP (partially observable Markov decision process) whose terminal state is compared against the goal state. To give credit for partial progress, ITBench introduces the Normalized Topology-Aware Match (NTAM), a topology-based distance score that rewards predicted root causes and fault-propagation chains that are close to the ground truth rather than only exactly equal.

What would settle it

The rates would be shown to measure benchmark choices rather than agent capability if, in a blinded trial, experienced SREs given the open-sourced scenarios also failed most of them or reached different root causes than the recorded ground truth; a cheaper check is to replay the 11 released scenarios on a different open-source application stack and see whether the model rankings and overall resolution rates survive the port.

Watch

Extended reading notes

Core claim

The paper's central claim is that current LLM agents are far from autonomous IT operations, and its evidence is measured success on ITBench's 94 scenarios: 13.8% for SRE incident resolution, 25.2% for CISO compliance assessment, and 0% for FinOps cost management, with no model mitigating a hard SRE scenario in any run and no model resolving a FinOps scenario at all. The authors attribute the gap largely to the fidelity of the testbeds — SRE scenarios reconstructed from 105 real incidents observed in their own SaaS products, CISO scenarios grounded in published CIS benchmarks, FinOps scenarios derived from FinOps Foundation business outcomes — rather than to flaws in the agents alone. They also report that the strongest agent is visibly non-deterministic across repeated runs on the same scenario, and that masking trace data cuts GPT-4o's SRE diagnosis rate from 18.10% to 9.52%, which they read as evidence that observability design is as decisive as model choice.

Load-bearing premise

The load-bearing premise is that ITBench's SRE scenarios capture real production IT difficulty: they are reconstructed from 105 incidents in the authors' own SaaS products, and because those incidents and the reconstruction are not released, a reader cannot verify that the reported resolution rates reflect the work SRE teams actually face.

Editorial extensions

If this is right

If these rates are representative, no tested agent — proprietary or open-weight — can be trusted for unsupervised production SRE, compliance, or cost management today.
Because masking trace data cuts GPT-4o's diagnosis rate almost in half, observability design is as important to agent performance as model choice, a lever the benchmark can now measure directly.
Success rates decline monotonically with the paper's complexity measure (the geometric mean of fault-propagation length, resolution steps, and technology count), so scenario difficulty can be tuned predictably and year-over-year agent progress tracked.
Because diagnosis and mitigation scores diverge — agents sometimes clear alerts while naming the wrong root cause — reporting each separately gives more actionable feedback than a single pass/fail number.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors leave implicit that these rates are an upper-bound for the tested agent family, not a property of all possible agents; a natural extension is to rerun the identical scenarios with specialized agents, such as ones with access to runbooks or historical incident databases.
A retrieval-augmented variant that feeds the agent past incident reports could plausibly raise SRE pass@1 before any new reasoning technique is attempted, since the paper's own examples show mitigation succeeding without correct diagnosis.
The FinOps result of 0% rests on only two scenarios, so that gap may reflect benchmark sparsity more than agent incapacity; adding cost-alert scenarios across more applications would settle whether FinOps is genuinely harder for agents.
The NTAM partial-credit idea could transfer to other agent benchmarks, where graded closeness to a ground-truth plan often gives better signal than binary pass/fail.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful part first: ITBench is the first benchmark I know of that puts SRE, CISO, and FinOps personas in live Kubernetes environments with automated, partial-credit evaluation. The NTAM topology-aware metric is a step past binary pass/fail on propagation chains, and the paper reports standard errors, run-to-run non-determinism, and trace on/off ablations. That is real evidence, and the framework is built to be extended. If you work on AIOps or agent evaluation, this is worth knowing about.

Now the soft spots. The stress-test note is correct: the abstract says 'resolve only 13.8% of SRE scenarios,' but Table 4 shows 13.81% is the diagnosis pass@1; mitigation pass@1 is 11.43%. 'Resolve' should mean the alert is cleared, so the headline overstates SRE capability and makes the cross-persona comparison apples-to-oranges (CISO's 25.2% is posture-assessment pass@1, FinOps's 0% is mitigation pass@1). That is a real mislabeling, but the qualitative conclusion survives: even the best agent clears alerts in only about one in nine SRE scenarios.

The bigger problem is verification. Only 11 of 94 scenarios are public, and the SRE scenarios come from 105 internal incidents at the authors' own SaaS products that are not released. The baseline agents and ground truths are authored by the same team. That does not make the benchmark useless, but it means the headline resolution rates are claims about a private test set, not an independently auditable result. I'd want at least a held-out public set, or a protocol for community audits, before treating the numbers as stable.

Minor issues: the 42 SRE scenarios are really 21 scenarios with trace on/off; the paper discloses this in a footnote but the abstract says '94 scenarios' without caveat. The complexity distribution is stated as Easy 24/Medium 52/Hard 24 in Figure 4b and Figure 9, but Appendix C.3.1 says Easy 24/Medium 24/Hard 52 — a typo that should be fixed. FinOps has two scenarios, so a 0% resolution rate is a data point, not a statement about FinOps capability.

Overall: this is a serious benchmark effort with a few load-bearing presentation problems, all fixable. The central finding — SOTA models are far from autonomous IT operations — holds up. I'd send it to peer review, but only with revisions that correct the metric labels, open up a meaningful subset, and clean up the counting and complexity errors.

Referee Report

4 major / 5 minor

Summary. ITBench is a framework and initial benchmark for evaluating LLM-based AI agents on IT automation tasks across three personas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The paper describes 94 scenarios, a runtime environment for push-button evaluation, three baseline agents, and a leaderboard. The main empirical claim, stated in the abstract and introduction, is that state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios, suggesting that current agents are far from autonomous IT operations.

Significance. If the numbers are reported correctly, ITBench would be a valuable addition to the growing set of agent benchmarks: it evaluates agents in a real Kubernetes-based environment with observability tools, reports multiple runs with standard errors, open-sources part of the framework and two baseline agents, and proposes interpretable partial-credit metrics (NTAM). The scale (94 advertised scenarios, 10 runs for SRE, 8 for CISO) and the explicit study of non-determinism are strengths. The central finding, that a capable proprietary model resolves a small fraction of realistic IT tasks, is timely and would motivate further research. However, the headline 'resolve' numbers conflate different task metrics, the scenario count is inflated, and the SRE ground truth is not publicly auditable; these issues must be corrected before the benchmark's claims can be accepted at face value.

major comments (4)

[Abstract; Section 4.2 (Tables 4, 5, 6)] The abstract and introduction report that agents 'resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios,' but these three numbers do not measure the same thing. For SRE, the 13.8% is the Diagnosis pass@1 for gpt-4o in Table 4, whereas the end-to-end 'resolve' metric in SRE terminology is Mitigation pass@1, which is 11.43% (or lower if a joint diagnosis-plus-mitigation metric is used). For CISO, 25.2%/24.74% is the overall posture-assessment pass@1 in Table 5, and for FinOps, 0% is the Mitigation pass@1 in Table 6 even though the Diagnosis pass@1 is 33%. The abstract's 'resolve' conflates diagnosis, posture assessment, and mitigation, overstating SRE capability and making the headline comparison ill-defined. Please report one consistently defined end-to-end success metric for all three personas, or relabel the abstract to state explicitly which task each number refers to.
[Section 3.1; Appendix C.6.4 (Tables 19-20)] The paper advertises '94 real-world scenarios,' but the 42 SRE scenarios are actually 21 fault patterns evaluated under two observability conditions (with and without tracing), as shown in Tables 19 and 20. These are not 42 distinct tasks; they are 21 scenarios measured twice. Similarly, the FinOps persona contains only 2 scenarios, so the claimed diversity across 94 scenarios is misleading. The paper should clearly distinguish scenario patterns from evaluation conditions, report the number of distinct tasks per persona, and avoid implying 94 unique real-world problems.
[Section 3.3; Section 4.1] The abstract's statement that 'agents powered by state-of-the-art models resolve only 13.8%...' is an overgeneralization because the evaluation only covers the authors' three baseline agents, which use specific tooling, prompts, and reflection strategies. The results are lower bounds on the performance of those particular baselines, not on the capabilities of GPT-4o or Llama as model families. The paper should either soften the abstract to attribute the numbers to 'the provided baseline agents' or explicitly acknowledge that other agent designs may achieve substantially different resolution rates.
[Section 3.4; Appendix C.3.1] The central 'real-world' claim for SRE rests on scenarios reconstructed from 105 incidents in the authors' own SaaS products, but neither the original incident reports nor the reconstruction mapping are released; only 11 of 94 scenarios are public. A reader cannot verify that the benchmark difficulty reflects production IT work rather than the authors' modeling choices. Please release the full scenario set, or at least a detailed mapping from each incident to its recreated scenario, the ground-truth fault chains, and the criteria used to decide which incidents were included. Without this, the SRE 'real-world' difficulty claim is not independently testable.

minor comments (5)

[Section 3.4] The paragraph beginning 'For each scenario that an agent works on...' is duplicated verbatim; please remove the repetition.
[Appendix C.6.3] NTAM is defined with configurable hyperparameters, but the paper does not report the hyperparameter values used in the experiments, so the partial-credit scores cannot be reproduced or compared across future submissions.
[Section 4.3; Figure 4] The Easy/Medium/Hard complexity thresholds are never defined numerically; the text refers to Equation (6) and domain knowledge, but a reader cannot determine why a scenario is 'Easy' versus 'Medium' from the reported factors alone.
[Table 6] The FinOps diagnosis and mitigation pass@1 values are reported as single percentages (0, 16.6, 33) without standard errors, unlike the SRE and CISO tables; please provide error bars or state the number of runs used.
[Appendix D.6.3; Table 24] There are typos in the appendix, including 'kyverno-upadate' in Table 24 and 'scneario_complexity' in Section 4.3; these should be corrected.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the reported agent success rates are direct empirical measurements; the headline mislabeling of diagnosis as 'resolve' is a reporting error, not a circular derivation.

full rationale

The paper's central claims are empirical benchmark measurements. The SRE resolution rate (13.8%) is the outcome of running the authors' own SRE-Agent (gpt-4o) against their 42 scenarios, and the value is read directly from Table 4's Diagnosis pass@1 column; the CISO and FinOps numbers likewise come from the corresponding evaluation tables. These are observations of agent behavior, not quantities derived by construction from the benchmark's definition. The scenario complexity measure (Eq. 6) is a labeling of scenarios, and the observed decline in pass@1 with complexity is a correlation, not a derivation of one from the other. No load-bearing self-citation chain exists: the authors' prior work is cited for background (fault injection, FinOps KPIs, common task framework), not to justify the benchmark's validity or to forbid alternative designs. The main concern is a metric-mislabeling issue: the abstract says agents 'resolve only 13.8% of SRE scenarios,' but Table 4 shows Diagnosis pass@1 = 13.81% while Mitigation pass@1 = 11.43%, and the CISO 25.2% is the posture-assessment pass@1 while the FinOps 0% is mitigation pass@1. This makes the headline comparison ill-defined and overstates SRE capability, but it is an inaccurate summary of empirical results rather than circularity: the numbers are not constructed to equal their inputs. Similarly, the real-world representativeness claim rests on the authors' reconstruction from their own SaaS incidents, which is an auditability/validity concern, not a reduction of the results to the benchmark's assumptions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claims rest on the representativeness of the scenarios (especially SRE, from internal incidents), the paper-specific complexity metric, the pass@1 ground truth, and the testbed fidelity. These are domain assumptions rather than standard math. The NTAM and complexity metrics have hand-set hyperparameters or thresholds, and the FinOps economics rely on the authors' custom pricing model.

free parameters (4)

NTAM component hyperparameters = not specified
Appendix C.6.3 defines topology distance, node importance, and length-mismatch penalties with tunable hyperparameters; no values are reported, and they influence all reported NTAM scores.
Scenario complexity thresholds for Easy/Medium/Hard = not specified
Equation (6) defines complexity as a geometric mean, but the cutoffs used to label scenarios Easy, Medium, or Hard are not stated; these labels drive the complexity analysis in Section 4.3.
FinOps custom pricing model rates = not specified
Appendix E.3 sets hourly CPU and memory cost rates plus negligible network/spot costs by hand to compute ground-truth optimal costs; values are not given, affecting FinOps proximity metrics.
Cost alert threshold = 20%
FinOps scenarios trigger alerts on a >20% cost increase (Tables 25 and 26); this chosen threshold defines scenario difficulty but is not externally anchored.

assumptions (4)

domain assumption SRE scenarios reconstructed from internal incidents are representative of real-world production incidents
Section 3.1 and Appendix C.3.1 state scenarios derive from 105 incidents in the authors' SaaS products; the source incidents are not released, so representativeness must be taken on trust.
ad hoc to paper Scenario complexity is adequately captured by the geometric mean in Eq. (6)
Equation (6) defines complexity from propagation path length, resolution steps, and technology count; this is a paper-specific definition with no external validation, and it is used to interpret performance drops.
domain assumption Exact match to author-authored ground truth is the correct success criterion (pass@1)
Appendix C.6.2 defines diagnosis and mitigation success as exact root-cause match or cleared alerts; the partial-credit NTAM is secondary, so the headline rates depend on this binary criterion.
domain assumption The Kubernetes/OpenTelemetry testbed faithfully reproduces production failure behavior
Appendix C.3.2 describes reconstruction of incidents in a synthetic cluster; the faithfulness of injected faults to real production faults is assumed.

invented entities (2)

Normalized Topology-Aware Match (NTAM)
purpose: Partial-credit scoring for fault localization and fault propagation chain evaluation
New metric defined in Appendix C.6.3 with tunable weights and no validation against human judgment or external benchmarks.
Scenario complexity score (Eq. 6)
purpose: Categorize scenarios into Easy/Medium/Hard and explain performance differences
Paper-specific definition using geometric mean of three factors; thresholds are unspecified and it is used as an explanatory variable in Section 4.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks." pith.science (2026). https://pith.science/paper/3Z2ODSFK

@misc{pith2026250205352,
  author       = {Pith},
  title        = {Pith review of: ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3Z2ODSFK}},
  note         = {Machine review of arXiv:2502.05352}
}

read the original abstract

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.

Figures

Figures reproduced from arXiv: 2502.05352 by the authors.

**Figure 1.** Sample personas and IT tasks. Bell icon represents event-triggered tasks. Information icon represents other tasks [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** ITBench automation framework. actions. Environment. E represents an an operational testbed where the problem occurs. Components within the environment expose APIs to observe and control the environment. When the Agent Builder registers the agent for benchmarking, the Benchmark Runner (see [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Agent and environment as a POMDP. Agents [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (23 more)

**Figure 4.** Figure 4: Characterization of ITBench scenarios. the formulation can be thought of as a POMDP: ot = h(st) (3) The set A of actions is defined as Q S {⊥}, where Q is the set of tools and ⊥ represents the ‘stop action’ by the agent. We define t ∗ as the time when agent stops: t ∗ …

**Figure 5.** Figure 5: ITBench leaderboard workflow. about a thousand individual machine failures and thousands of disk failures every year (Dean, 2009). Many of these failures further trigger correlated failures (Ford et al., 2010). New hardware fault models such as silent data corruptions …

**Figure 6.** Figure 6: Example ITBench leaderboard. testing). Detection is simplified with golden-signal-based alerts, which observability tools provide natively. Though, the challenge intensifies during an event storm, requiring SREs to distinguish actionable alerts by suppressing false pos…

**Figure 7.** Figure 7: Architecture of ITBench responsible for orchestrating SRE scenarios. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

**Figure 8.** Figure 8: Characterizing ITBench scenarios. Easy 24.0% Medium 52.0% Hard 24.0% [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

**Figure 9.** Figure 9: SRE scenario complexity. factor to skew the assessment. While factors like required skill sets or the number and type of diagnostic interactions (e.g., tool invocations or queries) could further refine our complexity measure, these factors are often highly dependent o…

**Figure 10.** Figure 10: Multi-modality data for SRE task. As shown in [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

**Figure 11.** Figure 11: SRE-Agent architecture deterministic results as possible. decoding_method is applicable for all models except gpt-4o [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]

**Figure 12.** Figure 12: Sample Trajectory of llama-3.3-70b-instruct in [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]

**Figure 13.** Figure 13: Sample Trajectory of gpt-4o in Scenario 19 [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]

**Figure 14.** Figure 14: Percent diagnosed for each scenario with tracing enabled. [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]

**Figure 15.** Figure 15: Normalized topology-aware metric (NTAM) for root cause for scenarios with tracing enabled. [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]

**Figure 16.** Figure 16: Normalized topology-aware metric (NTAM) for fault propagation chain (FPC) for each scenario with tracing [PITH_FULL_IMAGE:figures/full_fig_p035_16.png]

**Figure 17.** Figure 17: Percent repaired for each scenario with tracing enabled. [PITH_FULL_IMAGE:figures/full_fig_p036_17.png]

**Figure 18.** Figure 18: Example agent output for Scenario 15. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_18.png]

**Figure 19.** Figure 19: Compliance Authoring and Administration vs. Policy Validation Point Engines [PITH_FULL_IMAGE:figures/full_fig_p039_19.png]

**Figure 20.** Figure 20: Architecture of ITBench responsible for orchestrating CISO scenarios. [PITH_FULL_IMAGE:figures/full_fig_p041_20.png]

**Figure 21.** Figure 21: CISO scenario complexity. environment to measure the benchmarking metrics. Once the metrics for all predefined scenarios are collected, they are aggregated and displayed on the ITBench Leaderboard [PITH_FULL_IMAGE:figures/full_fig_p042_21.png]

**Figure 22.** Figure 22: CISO Compliance Assessment Agent end-to-end Benchmarking Process. [PITH_FULL_IMAGE:figures/full_fig_p043_22.png]

**Figure 23.** Figure 23: Sample Trajectory of LLaMa 405B in Success [PITH_FULL_IMAGE:figures/full_fig_p047_23.png]

**Figure 24.** Figure 24: Sample Trajectory of LLaMa 405B in Failure [PITH_FULL_IMAGE:figures/full_fig_p049_24.png]

**Figure 25.** Figure 25: Percent pass@1 for each scenario. 50 [PITH_FULL_IMAGE:figures/full_fig_p050_25.png]

**Figure 26.** Figure 26: Sample Trajectory of unusual cost variation use [PITH_FULL_IMAGE:figures/full_fig_p053_26.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Beyond Component Testing: Validating Agentic AI Systems
cs.AI 2026-07 conditional novelty 6.0 of 10

Agentic AI cannot be adequately validated by component tests alone; trajectory-in-context validation is required, and current practice is mature only for behavioral evaluation.
Motif 3: Technical Report
cs.AI 2026-08 conditional novelty 5.0 of 10

A 314B-parameter mixture-of-experts language model with grouped differential latent attention and multi-teacher distillation reports competitive results on agentic, coding, and reasoning benchmarks.
Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents
cs.AI 2026-05 conditional novelty 5.0 of 10

A synthetic scenario generation pipeline can expand an industrial agent benchmark to new asset classes with comparable quality and large runtime savings.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages · cited by 3 Pith papers

[1]

New benchmark jobs are stored in the Benchmark Queue for processing

work page
[2]

The Benchmark Runner fetches a benchmark scenario for a particular agent from the Benchmark Queue

work page
[3]

playground

The Benchmark Runner provisions the environment as per the benchmark scenario specification. The scenario’s environment is the set of systems required for the ex- ecution of a specific IT task.The Agent interacts with (and can potentially modify) the environment to solve the given IT automation tasks. A benchmark evaluation measures the Agent’s performanc...

work page
[4]

(b) Benchmark Runner deploys the scenario’s environ- ment by executing the deploy_scenario function

For each scenario included in the benchmark run, the Benchmark Runner and the Agent execute the following steps: (a) The Agent continuously polls theget_manifest API to monitor when a new manifest enters the Ready state. (b) Benchmark Runner deploys the scenario’s environ- ment by executing the deploy_scenario function. Each environment reports its status...

work page
[5]

node failure

Once the evaluation results for all the scenarios in the benchmark are ready, Benchmark Runner aggregates them and publishes the results to the Leaderboard. We instantiated the Leaderboard evaluation metrics for a few IT automation tasks as detailed in Section 3.1, Table 2. In Figure 6 shows the Leaderboard landing page displaying the benchmarking metrics...

work page 2002
[6]

Environment State: An environment has a countable set of states that we consider to mark a particular condition at a specific time. Example of states are an environment initial deployment state, an environment failure state 39 ITBench after a fault or non-compliant configuration injection, or an environment remediated or compliant state after mitigation

work page
[7]

pass" or

Goal: A goal is the desired state for the environment known as the goal state. Agents are tasked to transition environments from their initial state to their goal state in the most efficient manner. At their disposal are environment actions, including requests for observations or actuation attempts to affect the state of the system. The agents first step ...

work page 2024
[8]

Kick agent Y with report

Update leaderboard5. Kick agent Y with report

work page

Show all 21 references

[10]

We group the agents by Agent Types that reflect the IT operations personas, for example CISO, FInOps, or SRE type

Agent: An agent is an AI driven software that au- tonomously acts on behalf of a persona to solve a given task. We group the agents by Agent Types that reflect the IT operations personas, for example CISO, FInOps, or SRE type
[11]

Typi- cal tasks for CISO are to collect evidence and assess compliance controls posture

Task: A task is a specific job corresponding to the role of a persona that the agents aim to automate. Typi- cal tasks for CISO are to collect evidence and assess compliance controls posture
[12]

For CISO, for instance, each Kubernetes CIS-benchmark requirement instantiated on OPA is a unique scenario

Scenario: A scenario is a real-life occurrence of a task in a given setting. For CISO, for instance, each Kubernetes CIS-benchmark requirement instantiated on OPA is a unique scenario. The scenarios can be grouped in classes
[13]

Scenario Class: A scenario class is a class of real- life scenarios that are grouped together expecting the same behavior and outcome from the corresponding persona. Examples of scenarios classes are the set Kubernetes CIS-benchmarks on OPA engine, theset of RHEL9 CIS-benchmar...
[14]

Scenario Environment: A scenario environment is the part of the scenario that specifies the deployment set- tings
[17]

released a compliance as code SDK to support the machine readable compliance as code standard (OSCAL,
[18]

deploy_environment

of the National Institute of Standards and Technol- ogy for programmatic usage in compliance automation. ITBench CISO automation leverages this methodology to represent the CIS-benchmarks requirements, detect the events of creation or update of requirements, and trigger the cr...

2024
[20]

Clear deployed scenario2

Evaluate (success/fail, MTTR, etc.)7. Clear deployed scenario2. Deploy scenario X3. Inject a fault (issue)4. Find report
[21]

mini- mize the admission of containers wishing to share the host network namespace

Pick a scenario XFor every agent, do steps 1-8 for all relevant scenarios Figure 22: CISO Compliance Assessment Agent end-to-end Benchmarking Process. kubectl commands, is executed against a real Kubernetes cluster to collect evidence. The collected evidence is then assessed u...

2021
[365]

In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering, pages 1438–1443, 2021. G. Quattrocchi, E. Incerto, R. Pinciroli, C. Trubiani, and L. Baresi. Autoscaling solutions for cloud app...

2021 arXiv
[2015]

doi: 10.1145/2829988.2787496

ISSN 0146-4833. doi: 10.1145/2829988.2787496. Harvard Business Review Research Report. Tam- ing it complexity through effective strategies and partnerships. https://hbr.org/sponsored/ 2022/11/taming-it-complexity-through- effective-strategies-and-partnerships , 2022. P. H. Hoc...

2022
[2022]

have applied AI/ML techniques to speed up these tasks, focusing on mapping regulatory requirements to stan- dard control frameworks such as NIST 800-53 (NIST 800- 53). Our agentic automation in the current ITBench solution pioneers this type of effort to author compliance arti...

2023
[2024]

the AI revolution hasn’t happened yet

Accessed: 2025-01-30. O. Community. Opentelemetry astronomy shop demo. https://opentelemetry.io/docs/ demo/, 2024. Accessed: 2025-01-30. J. Dean. Designs, Lessons and Advice from Building Large Distributed Systems. In Proceedings of the the 3rd Large Scale Distributed Systems ...

2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

New benchmark jobs are stored in the Benchmark Queue for processing

work page

[2] [2]

The Benchmark Runner fetches a benchmark scenario for a particular agent from the Benchmark Queue

work page

[3] [3]

playground

The Benchmark Runner provisions the environment as per the benchmark scenario specification. The scenario’s environment is the set of systems required for the ex- ecution of a specific IT task.The Agent interacts with (and can potentially modify) the environment to solve the given IT automation tasks. A benchmark evaluation measures the Agent’s performanc...

work page

[4] [4]

(b) Benchmark Runner deploys the scenario’s environ- ment by executing the deploy_scenario function

For each scenario included in the benchmark run, the Benchmark Runner and the Agent execute the following steps: (a) The Agent continuously polls theget_manifest API to monitor when a new manifest enters the Ready state. (b) Benchmark Runner deploys the scenario’s environ- ment by executing the deploy_scenario function. Each environment reports its status...

work page

[5] [5]

node failure

Once the evaluation results for all the scenarios in the benchmark are ready, Benchmark Runner aggregates them and publishes the results to the Leaderboard. We instantiated the Leaderboard evaluation metrics for a few IT automation tasks as detailed in Section 3.1, Table 2. In Figure 6 shows the Leaderboard landing page displaying the benchmarking metrics...

work page 2002

[6] [6]

Environment State: An environment has a countable set of states that we consider to mark a particular condition at a specific time. Example of states are an environment initial deployment state, an environment failure state 39 ITBench after a fault or non-compliant configuration injection, or an environment remediated or compliant state after mitigation

work page

[7] [7]

pass" or

Goal: A goal is the desired state for the environment known as the goal state. Agents are tasked to transition environments from their initial state to their goal state in the most efficient manner. At their disposal are environment actions, including requests for observations or actuation attempts to affect the state of the system. The agents first step ...

work page 2024

[8] [8]

Kick agent Y with report

Update leaderboard5. Kick agent Y with report

work page