{"id":"abf96766-3b80-4b48-be2f-e5c28cf90f98","arxiv_id":"2502.05352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper presents ITBench, a benchmark for AI agents on IT operations tasks, and reports that today's agents resolve at most 25% of its scenarios.","lead":"This paper introduces ITBench, a benchmark framework for testing AI agents on IT operations tasks in three areas: site reliability, security compliance, and cloud cost management. In initial tests, even the best AI agents handled only a small fraction of scenarios, so the main near-term value is a shared yardstick for measuring progress.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 13.8% SRE 'resolution' is diagnosis pass@1, not scenario resolution; mitigation pass@1 is only 11.43%, so the headline metric is mislabeled.","rationale":"Single most load-bearing concern: the abstract's headline numbers are not consistently defined as 'resolution'. For SRE, the reported 13.8% is the diagnosis pass@1, not the mitigation/resolution rate; the actual mitigation pass@1 is 11.43%, and a joint diagnosis+mitigation success rate would be lower. The CISO 25.2% and FinOps 0% come from different task columns (overall posture assessment and mitigation, respectively). This makes the central claim quantitatively ill-defined: 'resolve' means different things in each persona, and the SRE number overstates the agent's ability to actually fix incidents. The reader's concern about SRE scenario representativeness is real, but it concerns external validity; the metric conflation is an internal correctness issue that can be checked directly from the paper's tables. Even if the scenarios are perfectly representative, the headline as written misreports what was measured. Since the fix is a relabeling or a recomputed joint metric, the appropriate verdict remains conditional: the framework and open-sourced agents are valuable, but the central quantitative claim needs correction before the paper can be accepted as stated.","tokens_in":49666,"tokens_out":7709,"duration_ms":70021,"concrete_test":"Recompute from the paper's own tables: locate Table 4 and confirm that 13.81% is in the 'Diagnosis pass@1' column and 11.43% in the 'Mitigation pass@1' column for gpt-4o. Then derive an end-to-end SRE 'resolution' rate as either (a) mitigation pass@1 (11.43%) or (b) the fraction of scenario runs where both diagnosis and mitigation pass (using Tables 17-18), and compare to the abstract's 13.8%. If the corrected rate is below 13.8%, the headline overstates performance. Also check Tables 5 and 6 to verify which column the CISO 25.2% and FinOps 0% come from.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 (Table 4) reports for the best SRE agent (gpt-4o) a Diagnosis pass@1 of 13.81% and a Mitigation pass@1 of 11.43%. The abstract and introduction describe 'resolve only 13.8% of SRE scenarios' and a 'successful scenario handling rate of 13.8%'. In SRE terminology, resolving an incident means mitigation (clearing the alert), not merely diagnosis. The 13.8% figure matches the diagnosis column, not the mitigation column. The same metric mismatch appears across personas: the CISO 25.2% matches the overall posture-assessment pass@1 in Table 5 (gpt-4o row, 24.74±2.64), while the FinOps 0% matches the mitigation pass@1 in Table 6 (gpt-4o mitigation 0%, but diagnosis 33%). Thus the three headline numbers are not the same kind of measurement; 'resolve' conflates diagnosis, posture assessment, and mitigation. If 'resolve' means completing the end-to-end task, the correct SRE number is 11.43% (or a joint diagnosis+mitigation metric, which is lower). This mislabeling overstates SRE capability and makes the headline comparison ill-defined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ITBench is a framework and initial benchmark for evaluating LLM-based AI agents on IT automation tasks across three personas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The paper describes 94 scenarios, a runtime environment for push-button evaluation, three baseline agents, and a leaderboard. The main empirical claim, stated in the abstract and introduction, is that state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios, suggesting that current agents are far from autonomous IT operations.","tokens_in":49936,"tokens_out":3810,"duration_ms":38477,"significance":"If the numbers are reported correctly, ITBench would be a valuable addition to the growing set of agent benchmarks: it evaluates agents in a real Kubernetes-based environment with observability tools, reports multiple runs with standard errors, open-sources part of the framework and two baseline agents, and proposes interpretable partial-credit metrics (NTAM). The scale (94 advertised scenarios, 10 runs for SRE, 8 for CISO) and the explicit study of non-determinism are strengths. The central finding, that a capable proprietary model resolves a small fraction of realistic IT tasks, is timely and would motivate further research. However, the headline 'resolve' numbers conflate different task metrics, the scenario count is inflated, and the SRE ground truth is not publicly auditable; these issues must be corrected before the benchmark's claims can be accepted at face value.","major_comments":[{"comment":"The abstract and introduction report that agents 'resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios,' but these three numbers do not measure the same thing. For SRE, the 13.8% is the Diagnosis pass@1 for gpt-4o in Table 4, whereas the end-to-end 'resolve' metric in SRE terminology is Mitigation pass@1, which is 11.43% (or lower if a joint diagnosis-plus-mitigation metric is used). For CISO, 25.2%/24.74% is the overall posture-assessment pass@1 in Table 5, and for FinOps, 0% is the Mitigation pass@1 in Table 6 even though the Diagnosis pass@1 is 33%. The abstract's 'resolve' conflates diagnosis, posture assessment, and mitigation, overstating SRE capability and making the headline comparison ill-defined. Please report one consistently defined end-to-end success metric for all three personas, or relabel the abstract to state explicitly which task each number refers to.","section":"Abstract; Section 4.2 (Tables 4, 5, 6)"},{"comment":"The paper advertises '94 real-world scenarios,' but the 42 SRE scenarios are actually 21 fault patterns evaluated under two observability conditions (with and without tracing), as shown in Tables 19 and 20. These are not 42 distinct tasks; they are 21 scenarios measured twice. Similarly, the FinOps persona contains only 2 scenarios, so the claimed diversity across 94 scenarios is misleading. The paper should clearly distinguish scenario patterns from evaluation conditions, report the number of distinct tasks per persona, and avoid implying 94 unique real-world problems.","section":"Section 3.1; Appendix C.6.4 (Tables 19-20)"},{"comment":"The abstract's statement that 'agents powered by state-of-the-art models resolve only 13.8%...' is an overgeneralization because the evaluation only covers the authors' three baseline agents, which use specific tooling, prompts, and reflection strategies. The results are lower bounds on the performance of those particular baselines, not on the capabilities of GPT-4o or Llama as model families. The paper should either soften the abstract to attribute the numbers to 'the provided baseline agents' or explicitly acknowledge that other agent designs may achieve substantially different resolution rates.","section":"Section 3.3; Section 4.1"},{"comment":"The central 'real-world' claim for SRE rests on scenarios reconstructed from 105 incidents in the authors' own SaaS products, but neither the original incident reports nor the reconstruction mapping are released; only 11 of 94 scenarios are public. A reader cannot verify that the benchmark difficulty reflects production IT work rather than the authors' modeling choices. Please release the full scenario set, or at least a detailed mapping from each incident to its recreated scenario, the ground-truth fault chains, and the criteria used to decide which incidents were included. Without this, the SRE 'real-world' difficulty claim is not independently testable.","section":"Section 3.4; Appendix C.3.1"}],"minor_comments":[{"comment":"The paragraph beginning 'For each scenario that an agent works on...' is duplicated verbatim; please remove the repetition.","section":"Section 3.4"},{"comment":"NTAM is defined with configurable hyperparameters, but the paper does not report the hyperparameter values used in the experiments, so the partial-credit scores cannot be reproduced or compared across future submissions.","section":"Appendix C.6.3"},{"comment":"The Easy/Medium/Hard complexity thresholds are never defined numerically; the text refers to Equation (6) and domain knowledge, but a reader cannot determine why a scenario is 'Easy' versus 'Medium' from the reported factors alone.","section":"Section 4.3; Figure 4"},{"comment":"The FinOps diagnosis and mitigation pass@1 values are reported as single percentages (0, 16.6, 33) without standard errors, unlike the SRE and CISO tables; please provide error bars or state the number of runs used.","section":"Table 6"},{"comment":"There are typos in the appendix, including 'kyverno-upadate' in Table 24 and 'scneario_complexity' in Section 4.3; these should be corrected.","section":"Appendix D.6.3; Table 24"}],"recommendation":"major_revision","confidential_remarks":"The metric-conflation issue in the abstract is the most serious problem because it directly affects the headline claim; fortunately it is fixable by re-reporting. The scenario-count inflation and the closed SRE ground truth also need to be addressed. The paper is a benchmark contribution, not a theory paper, so the small number of FinOps scenarios (2) and the fact that the authors wrote both the baselines and the ground truth are acceptable if disclosed clearly, but the current framing overstates coverage and novelty. The journal should require the authors to either release more scenarios or substantially tone down the 'real-world' generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part first: ITBench is the first benchmark I know of that puts SRE, CISO, and FinOps personas in live Kubernetes environments with automated, partial-credit evaluation. The NTAM topology-aware metric is a step past binary pass/fail on propagation chains, and the paper reports standard errors, run-to-run non-determinism, and trace on/off ablations. That is real evidence, and the framework is built to be extended. If you work on AIOps or agent evaluation, this is worth knowing about.\n\nNow the soft spots. The stress-test note is correct: the abstract says 'resolve only 13.8% of SRE scenarios,' but Table 4 shows 13.81% is the diagnosis pass@1; mitigation pass@1 is 11.43%. 'Resolve' should mean the alert is cleared, so the headline overstates SRE capability and makes the cross-persona comparison apples-to-oranges (CISO's 25.2% is posture-assessment pass@1, FinOps's 0% is mitigation pass@1). That is a real mislabeling, but the qualitative conclusion survives: even the best agent clears alerts in only about one in nine SRE scenarios.\n\nThe bigger problem is verification. Only 11 of 94 scenarios are public, and the SRE scenarios come from 105 internal incidents at the authors' own SaaS products that are not released. The baseline agents and ground truths are authored by the same team. That does not make the benchmark useless, but it means the headline resolution rates are claims about a private test set, not an independently auditable result. I'd want at least a held-out public set, or a protocol for community audits, before treating the numbers as stable.\n\nMinor issues: the 42 SRE scenarios are really 21 scenarios with trace on/off; the paper discloses this in a footnote but the abstract says '94 scenarios' without caveat. The complexity distribution is stated as Easy 24/Medium 52/Hard 24 in Figure 4b and Figure 9, but Appendix C.3.1 says Easy 24/Medium 24/Hard 52 — a typo that should be fixed. FinOps has two scenarios, so a 0% resolution rate is a data point, not a statement about FinOps capability.\n\nOverall: this is a serious benchmark effort with a few load-bearing presentation problems, all fixable. The central finding — SOTA models are far from autonomous IT operations — holds up. I'd send it to peer review, but only with revisions that correct the metric labels, open up a meaningful subset, and clean up the counting and complexity errors.","headline":"A genuinely useful multi-persona IT benchmark with a fixable metric-labeling problem and too little public data to verify the headline numbers.","tokens_in":50799,"tokens_out":4243,"would_cite":true,"duration_ms":39524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents ITBench, a benchmark that tests AI agents on real-world IT automation tasks, and shows that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps…","keywords":["AI agents","IT automation","benchmarking","LLM evaluation","Site Reliability Engineering","compliance assessment","FinOps","incident resolution"],"falsifier":"The rates would be shown to measure benchmark choices rather than agent capability if, in a blinded trial, experienced SREs given the open-sourced scenarios also failed most of them or reached different root causes than the recorded ground truth; a cheaper check is to replay the 11 released scenarios on a different open-source application stack and see whether the model rankings and overall resolution rates survive the port.","tokens_in":49455,"feed_emoji":"🤖","tokens_out":15492,"duration_ms":125197,"temperature":0.7,"pith_summary":"ITBench is an open benchmarking framework that measures whether AI agents can automate real-world IT operations across three personas: Site Reliability Engineering (diagnosing and mitigating production incidents), CISO compliance (translating security-control requirements into executable policy and checking posture), and FinOps (diagnosing and fixing cloud cost anomalies). The paper's central finding is that agents built on state-of-the-art LLMs resolve only 13.8% of 42 SRE scenarios, 25.2% of 50 CISO scenarios, and 0% of 2 FinOps scenarios, with GPT-4o far ahead of open-weight models but still far from reliable. The authors argue that fully autonomous IT automation is not yet achievable and that progress needs a standardized, extensible benchmark with automated partial-credit evaluation. ITBench formalizes each task as a scenario (a tuple of metadata, a live testbed environment, triggering alerts, and a desired outcome) and scores agents on both exact success and topology-aware partial credit.","feed_headline":"Top AI agents resolve just 13.8% of real IT incidents in benchmark","feed_subtitle":"New benchmark finds LLM agents far from autonomous on incident, compliance, and cost tasks.","key_machinery":"The carrying mechanism is the scenario specification, formalized as a tuple $\\langle M, E, T, D\\rangle$: $M$ is the metadata and ground truth, $E$ is an operational testbed (a Kubernetes cluster running a demo application with Grafana, Prometheus, Loki, and Jaeger observability), $T$ is a set of golden-signal triggering alerts, and $D$ is the automation goal. The agent interacts with $E$ through natural-language tools (NL2Kubectl, NL2Traces, NL2Metrics, NL2Logs, NL2Alerts), and the interaction is modelled as a POMDP (partially observable Markov decision process) whose terminal state is compared against the goal state. To give credit for partial progress, ITBench introduces the Normalized Topology-Aware Match (NTAM), a topology-based distance score that rewards predicted root causes and fault-propagation chains that are close to the ground truth rather than only exactly equal.","core_discovery":"The paper's central claim is that current LLM agents are far from autonomous IT operations, and its evidence is measured success on ITBench's 94 scenarios: 13.8% for SRE incident resolution, 25.2% for CISO compliance assessment, and 0% for FinOps cost management, with no model mitigating a hard SRE scenario in any run and no model resolving a FinOps scenario at all. The authors attribute the gap largely to the fidelity of the testbeds — SRE scenarios reconstructed from 105 real incidents observed in their own SaaS products, CISO scenarios grounded in published CIS benchmarks, FinOps scenarios derived from FinOps Foundation business outcomes — rather than to flaws in the agents alone. They also report that the strongest agent is visibly non-deterministic across repeated runs on the same scenario, and that masking trace data cuts GPT-4o's SRE diagnosis rate from 18.10% to 9.52%, which they read as evidence that observability design is as decisive as model choice.","pith_inferences":["The authors leave implicit that these rates are an upper-bound for the tested agent family, not a property of all possible agents; a natural extension is to rerun the identical scenarios with specialized agents, such as ones with access to runbooks or historical incident databases.","A retrieval-augmented variant that feeds the agent past incident reports could plausibly raise SRE pass@1 before any new reasoning technique is attempted, since the paper's own examples show mitigation succeeding without correct diagnosis.","The FinOps result of 0% rests on only two scenarios, so that gap may reflect benchmark sparsity more than agent incapacity; adding cost-alert scenarios across more applications would settle whether FinOps is genuinely harder for agents.","The NTAM partial-credit idea could transfer to other agent benchmarks, where graded closeness to a ground-truth plan often gives better signal than binary pass/fail."],"forward_implications":["If these rates are representative, no tested agent — proprietary or open-weight — can be trusted for unsupervised production SRE, compliance, or cost management today.","Because masking trace data cuts GPT-4o's diagnosis rate almost in half, observability design is as important to agent performance as model choice, a lever the benchmark can now measure directly.","Success rates decline monotonically with the paper's complexity measure (the geometric mean of fault-propagation length, resolution steps, and technology count), so scenario difficulty can be tuned predictably and year-over-year agent progress tracked.","Because diagnosis and mitigation scores diverge — agents sometimes clear alerts while naming the wrong root cause — reporting each separately gives more actionable feedback than a single pass/fail number."],"supporting_citations":[{"why":"SWE-bench, the software-engineering agent benchmark the authors explicitly aim to emulate; supplies the benchmark paradigm ITBench is positioned against.","marker":"(Jimenez et al., 2024)"},{"why":"AIOpsLab, the closest existing SRE-agent benchmark, whose real-environment approach ITBench extends with multi-persona coverage, trace masking, and a verified leaderboard.","marker":"(Chen et al., 2024a)"},{"why":"ReAct, the reasoning-and-acting pattern that drives the baseline SRE, CISO, and FinOps agents whose success rates are the paper's central measurement.","marker":"(Yao et al., 2023)"},{"why":"Reflexion, the reflection mechanism the baseline agents use for linting and self-correction of generated commands and policies.","marker":"(Shinn et al., 2023)"},{"why":"CIS Benchmarks, the published security guidelines that define the compliance ground truth for all 50 CISO scenarios.","marker":"(CIS, 2024)"},{"why":"FinOps Foundation benchmarks, the business-outcome definitions from which the two FinOps scenarios are derived.","marker":"(Foundation, 2025a)"},{"why":"The pass@k estimator (from the Codex paper) that defines how the headline scenario success rates are computed.","marker":"(Chen et al., 2021)"}],"fun_headline_variants":["AI agents resolve 13.8% SRE, 25.2% CISO, 0% FinOps","ITBench: AI agents solve zero FinOps cost scenarios","LLM agents fail 86% of SRE incidents in ITBench","ITBench reveals AI agents far from autonomous IT ops","AI agents succeed on a quarter of compliance, none of cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ITBench's SRE scenarios capture real production IT difficulty: they are reconstructed from 105 incidents in the authors' own SaaS products, and because those incidents and the reconstruction are not released, a reader cannot verify that the reported resolution rates reflect the work SRE teams actually face.","fun_headline_variants_meta":{"raw":{"variants":["AI agents resolve 13.8% SRE, 25.2% CISO, 0% FinOps","ITBench: AI agents solve zero FinOps cost scenarios","LLM agents fail 86% of SRE incidents in ITBench","ITBench reveals AI agents far from autonomous IT ops","AI agents succeed on a quarter of compliance, none of cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001282,"raw_usage":{"total_tokens":5224,"prompt_tokens":917,"completion_tokens":4307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":4220}},"tokens_in":533,"tokens_out":4307,"duration_ms":29626,"temperature":1.0,"reasoning_tokens":4220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:40:39.026302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The rates would be shown to measure benchmark choices rather than agent capability if, in a blinded trial, experienced SREs given the open-sourced scenarios also failed most of them or reached different root causes than the recorded ground truth; a cheaper check is to replay the 11 released scenarios on a different open-source application stack and see whether the model rankings and overall resolution rates survive the port.","supporting_citations":[],"review_version":1}