{"id":"868bf120-4123-421c-97a0-8ebe02dba44d","arxiv_id":"2606.27154","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenRCA 2.0 is the first cross-system RCA benchmark with step-wise causal annotations, revealing that 11 frontier LLMs achieve 20.7% exact root-cause recovery and struggle with causal grounding (61.5% vs 76.0% ungrounded).","lead":"The paper introduces PAVE, a labeling protocol that uses fault injections to annotate full causal propagation paths in root cause analysis datasets rather than just final outcomes. This creates OpenRCA 2.0, a benchmark showing that frontier LLMs recover exact root causes in only about 21% of cases and often fail to ground them in verified causal paths.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"PAVE protocol's forward verification via fault injection may miss links or introduce labeling errors, undermining the 'ungrounded diagnosis' metric","rationale":"The reader's weakest_assumption directly identifies the same methodological hinge. Because the original review had only the abstract, the full text would need to supply explicit validation of PAVE (e.g., inter-expert agreement or sensitivity analysis on intervention choice) before the causal-grounding claim can be treated as robust. No other internal inconsistency appears in the reported numbers themselves.","tokens_in":1740,"tokens_out":332,"duration_ms":17216,"concrete_test":"Select 30 random OpenRCA 2.0 instances; have two independent system experts (blind to PAVE labels) reconstruct causal paths from logs and intervention records; compute inter-rater agreement with PAVE and the fraction of PAVE paths that experts flag as incomplete or incorrect. If agreement <75% or >15% of paths are disputed, the 61.5% figure cannot be interpreted as evidence of ungrounded reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance gap (76.0% identify correct service vs 61.5% ground it in a verified path) rests on PAVE producing reliable step-wise causal annotations. PAVE reconstructs paths by applying known interventions and tracing forward from cause to symptom. This assumes interventions are exhaustive, that no confounding paths exist, and that the resulting labels contain neither omissions nor false links. If any of those fail, the relaxed criterion no longer isolates a genuine failure mode distinct from outcome-only evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper argues that existing RCA benchmarks label only root causes rather than causal propagation paths, reducing evaluation to pattern matching. It introduces the PAVE protocol, which uses forward verification via known fault-injection interventions to reconstruct step-wise causal paths, yielding the OpenRCA 2.0 benchmark of 500 cross-system instances with these annotations. Evaluation across 11 frontier LLMs reports 20.7% exact root-cause set recovery on average; relaxing the criterion shows 76.0% of agents identify at least one correct root-cause service while only 61.5% ground it in a verified causal path to the symptom.","tokens_in":1861,"tokens_out":428,"duration_ms":19363,"significance":"If the PAVE annotations prove reliable, the distinction between service identification and path-grounded diagnosis identifies a concrete failure mode that outcome-only metrics obscure, providing a clearer signal for improving LLM agent reasoning and tool use in RCA. The work supplies the first publicly described step-wise causal benchmark in this domain.","major_comments":[{"comment":"PAVE protocol description: the central performance gap (76.0% vs 61.5%) rests on the claim that fault-injection interventions produce exhaustive, error-free causal paths; however, the manuscript provides no inter-annotator agreement, error analysis, or validation against confounding paths for the 500 instances, which directly undermines the reliability of the 'ungrounded diagnosis' metric.","section":"PAVE protocol description"},{"comment":"Dataset construction (500 instances): the abstract states the benchmark is 'cross-system' but supplies no criteria for instance selection, system diversity, or how interventions were chosen to ensure the reconstructed paths are representative rather than artifactual.","section":"Dataset construction"}],"minor_comments":[{"comment":"The abstract reports aggregate percentages without per-model breakdowns or confidence intervals, which would help readers assess variability across the 11 LLMs.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments highlighting the importance of validating the PAVE protocol and clarifying dataset construction. We respond point by point below, defending the objective nature of our intervention-based annotations while agreeing to strengthen the manuscript where gaps exist.","responses":[{"response":"PAVE reconstructs paths via deterministic forward verification from known fault-injection interventions rather than subjective human labeling, rendering traditional inter-annotator agreement inapplicable. The paths are verifiable by construction through the interventions themselves. We nevertheless agree that an explicit error analysis and discussion of potential confounding paths would strengthen the reliability claim for the ungrounded diagnosis metric and will add this to the revised manuscript.","revision_made":"partial","referee_comment":"[PAVE protocol description] PAVE protocol description: the central performance gap (76.0% vs 61.5%) rests on the claim that fault-injection interventions produce exhaustive, error-free causal paths; however, the manuscript provides no inter-annotator agreement, error analysis, or validation against confounding paths for the 500 instances, which directly undermines the reliability of the 'ungrounded diagnosis' metric."},{"response":"We will expand the manuscript with a dedicated section detailing the instance selection criteria, quantitative measures of system diversity across the 500 cross-system cases, and the rationale for choosing interventions to ensure the causal paths reflect representative RCA scenarios rather than artifacts.","revision_made":"yes","referee_comment":"[Dataset construction] Dataset construction (500 instances): the abstract states the benchmark is 'cross-system' but supplies no criteria for instance selection, system diversity, or how interventions were chosen to ensure the reconstructed paths are representative rather than artifactual."}],"tokens_in":1412,"tokens_out":370,"duration_ms":26975,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main things to know are that this work releases OpenRCA 2.0, a 500-instance benchmark with cross-system causal propagation paths, and that it finds frontier LLMs recover the exact root-cause set only 20.7 percent of the time on average. They also relax the metric and report that models name at least one correct root-cause service 76 percent of the time but ground it in a verified causal path only 61.5 percent of the time.\n\nWhat the paper does reasonably is flag a concrete gap in current evaluation. Outcome-only labels let models get credit for pattern matching without showing they understand how the fault actually propagated. The forward-verification idea in PAVE is a clean way to try to close that gap, and the numbers across 11 models give a usable baseline for people working on agentic debugging.\n\nThe soft spot is the thin evidence on whether PAVE itself is reliable. The protocol reconstructs paths by applying known interventions and tracing forward, but the abstract supplies no inter-annotator agreement, no error analysis on the 500 cases, and no checks for missed links or confounding paths. If the fault-injection traces are incomplete or noisy, the 76-to-61.5 gap no longer cleanly isolates the claimed failure mode. That assumption is load-bearing and needs explicit support.\n\nThis is for groups building or benchmarking LLM agents on operations tasks. A reader who cares about causal supervision in tool-use settings will find the dataset and the relaxed metric worth looking at. The paper deserves a serious referee because the core distinction it draws is worth testing even if the current methods section needs more detail on labeling quality.","headline":"The paper gives a new RCA benchmark with step-wise causal paths via PAVE and shows LLMs often spot a root service without tracing the actual chain, but the labeling protocol lacks reported validation.","tokens_in":2364,"tokens_out":418,"would_cite":false,"duration_ms":25390,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLM agents identify correct root causes but verify their causal paths to symptoms in only 61.5 percent of cases.","keywords":["root cause analysis","LLM agents","causal propagation paths","fault injection","process supervision","benchmark","causal annotations"],"falsifier":"Independent measurement of causal links in a production system that shows systematic mismatches with the paths reconstructed by the PAVE protocol on the same injected faults.","tokens_in":2666,"feed_emoji":"🔗","tokens_out":693,"duration_ms":27750,"temperature":0.7,"pith_summary":"Existing root cause analysis datasets for LLMs label only the final root cause service without the propagation path that connects it to the observed symptom. This design reduces evaluation to pattern matching instead of testing whether agents can trace verified causal chains. The paper introduces the PAVE protocol, which reconstructs those paths by applying known fault injection interventions and verifying forward from cause to effect. The resulting OpenRCA 2.0 benchmark contains 500 cross-system instances with step-wise causal annotations. Across 11 frontier models, exact root-cause set recovery averages 20.7 percent while the gap between spotting a correct service and grounding it in a verified path reaches 14.5 points.","feed_headline":"LLM root cause agents ground diagnoses in causal paths 61.5% of the time","feed_subtitle":"New step-wise benchmark built from fault injections shows outcome labels alone hide unverified propagation failures.","key_machinery":"The PAVE protocol, which reconstructs causal propagation paths by applying known fault-injection interventions and performing forward verification from cause to effect rather than backward inference from symptoms.","core_discovery":"By replacing outcome-only labels with step-wise causal process supervision, OpenRCA 2.0 shows that frontier LLMs recover the exact root-cause set in only 20.7 percent of cases on average. Agents locate at least one correct root-cause service in 76.0 percent of cases yet succeed in grounding that service inside a verified causal propagation path to the symptom in just 61.5 percent of cases. Outcome-only scoring therefore conceals the ungrounded diagnosis failure mode that step-wise annotations expose.","pith_inferences":["The same gap between identifying an answer and verifying its causal justification may appear in other multi-step agent tasks that involve tracing effects through complex systems.","Benchmarks limited to outcome labels may systematically overestimate agent performance whenever the domain requires explicit causal chaining.","The forward-verification approach could be adapted to label propagation steps in domains such as software debugging or medical differential diagnosis."],"forward_implications":["Exact recovery of the full root-cause set succeeds in only 20.7 percent of cases across frontier models.","Agents reach a correct root-cause service in 76.0 percent of cases but ground that service in a verified causal path in only 61.5 percent.","Outcome-only evaluation masks the ungrounded diagnosis failure mode that step-wise labels reveal.","Causal process supervision is required to build trustworthy LLM-based root cause analysis agents."],"fun_headline_variants":["Exact root cause recovery by LLMs at 20.7 percent average","Root cause service identified by agents in 76 percent of cases","Causal path grounding at 61.5 percent for LLM agents","Outcome labels hide ungrounded LLM root cause diagnoses"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Known interventions from fault injection can accurately reconstruct the true causal propagation paths without introducing labeling errors or missing links.","fun_headline_variants_meta":{"raw":{"variants":["Exact root cause recovery by LLMs at 20.7 percent average","Root cause service identified by agents in 76 percent of cases","Causal path grounding at 61.5 percent for LLM agents","Outcome labels hide ungrounded LLM root cause diagnoses"]},"model":"grok-4.3","cost_usd":0.006784,"raw_usage":{"total_tokens":3097,"prompt_tokens":713,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":67840500,"prompt_tokens_details":{"text_tokens":713,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2314,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":713,"tokens_out":70,"duration_ms":22111,"temperature":1.0,"reasoning_tokens":2314,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T07:02:22.813166+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent measurement of causal links in a production system that shows systematic mismatches with the paths reconstructed by the PAVE protocol on the same injected faults.","supporting_citations":[],"review_version":2}