Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM

(2) Harmonic Inc; (3) Peng Cheng Laboratory; (4) Unaffiliated); Charley Peter Chen (2); Hu Li (4); Liyi Zeng (3); Wei Xu (1) ((1) Tsinghua University; Wei Yin (1); Xin Liu (1); Yong Xiang (1)

REVIEW 4 major objections 5 minor 69 references

Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SynergyRCA claims that an off-the-shelf LLM, guided by a graph of Kubernetes runtime states and expert prompts, identifies incident root causes in about two minutes with roughly 90 percent precision.

desk verdict A coherent and useful RCA architecture, but the headline precision is not reproducible until the undefined snapshot-tolerance adjustment is specified and artifacts are released. read the letter →

arxiv 2506.02490 v1 pith:5A7HCCTC submitted 2025-06-03 cs.DC cs.AIcs.SE

Yong Xiang (1) , Charley Peter Chen (2) , Liyi Zeng (3) , Wei Yin (1) , Xin Liu (1) , Hu Li (4) , Wei Xu (1) ((1) Tsinghua University , (2) Harmonic Inc

show 2 more authors

(3) Peng Cheng Laboratory (4) Unaffiliated)

This is my paper

classification cs.DCcs.AIcs.SE

keywords RootCauseAnalysisKubernetesLargeLanguageModelsRetrieval-AugmentedGenerationStateGraphDatabaseIncidentManagementReconciliation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SynergyRCA claims that a production Kubernetes team can diagnose the root cause of an incident in roughly two minutes, with a precision around 0.90, by asking an off-the-shelf LLM to reason over a graph of the cluster's runtime states rather than over raw logs or a fixed rule set. The paper argues that the key is to build a StateGraph that records what each entity (Pods, PersistentVolumeClaims, NFS directories, quotas) looked like over time, plus a MetaGraph that records which kinds of entities can be connected. When an error arrives, the LLM predicts which resource kind is most likely responsible, the graphs supply a concrete path of connected entities, and a state-checking LLM verifies each entity's recorded state against three state-reconciliation rules. The tool is evaluated on one week and one six-month dataset from two production clusters and is said to identify 18 and 20 root-cause types, including five types the authors had not pre-specified. The significance is that accurate RCA is achieved without fine-tuning the LLM, which is costly and hard to keep current as Kubernetes versions and configurations change.

What carries the argument

The load-bearing object is the StateGraph, a graph database whose vertices are Kubernetes entities (Pod, PVC, Namespace, NFS directory) and their periodic snapshots, with edges labelled ReferInternal and UseExternal for entity references and HasState and HasEvent for snapshots. A MetaGraph is collapsed from it, keeping one vertex per entity kind and the same edge types, so the LLM can be guided along valid metapaths instead of inventing impossible connections. The workflow matches an incident to a source entity kind, asks the LLM in Triage to predict the destination kind, converts a metapath into a Cypher query, retrieves a concrete statepath, and uses StateChecker and ReportGen to compare each entity's snapshot content against the error message using three state-reconciliation rules. The machinery matters because it prevents the LLM from hallucinating relationships and keeps direct LLM access away from the production cluster.

What would settle it

Run SynergyRCA on a test cluster with a Pod that fails to schedule because its PersistentVolumeClaim is not bound, while snapshots are collected every five minutes, and check the generated report for an accurate 'Unbound' explanation; the paper's own data predicts precision near 0.26 to 0.46 for this class. A stronger test is to accelerate snapshot capture for PVC states and see whether that class's precision rises to the claimed 0.90 average.

Watch

Extended reading notes

Core claim

The paper's central claim is that structuring Kubernetes' runtime state as a graph, and letting an LLM navigate that graph with expert prompts, converts incident triage into a state-consistency check: verify the existence of the current state, verify its correctness, and identify discrepancies among states. The paper reports that this yields a correct root-cause explanation for 88% of sampled error messages in the first cluster and 92% in the second, at an average latency around two minutes, with remediation commands included. It also reports that the approach finds root-cause types that were not in the predefined taxonomy, because the LLM can propose novel kinds and the graph provides the evidence to confirm them.

Load-bearing premise

The five-minute periodic snapshots must capture enough of the cluster's changing state for the comparison of current versus desired state to reflect the incident; the paper itself reports that this breaks for fast-changing resources such as volume-binding states.

Editorial extensions

If this is right

Incident response in a Kubernetes cluster can move from log-grep to an end-to-end pipeline that outputs a root-cause statement and a remediation command in about two minutes per attempt.
Root-cause analysis no longer needs per-cluster fine-tuning of the LLM; graph construction and prompt customization carry the cluster-specific knowledge.
Previously unseen root-cause types can be surfaced, because the LLM proposes new destination kinds and the graph supplies the supporting evidence for them.
The cost per diagnosis is bounded: roughly 73K to 161K tokens, about $0.19 to $0.41 per attempt with GPT-4o pricing, with almost all cost in input prompts.
Fast-changing resources are the known weak point; if snapshots miss the transitional state, precision drops to 0.26 to 0.46 for cases like UnboundPVC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to replace the fixed five-minute polling with event-driven or post-hoc snapshot alignment for fast-changing resources; the paper's own numbers suggest this would lift the UnboundPVC class toward the overall 0.90 average.
The state-reconciliation framing is not Kubernetes-specific: any controller-based system that compares desired and current state could be wrapped in a StateGraph, so the same tool could plausibly diagnose Terraform, operator, or edge-device reconciliation loops if the graphs are constructed from their runtime stores.
Because almost all token cost is prompt input from StateChecker, optimizing which STATE vertices and which JSON fields are sent, rather than reducing the number of LLM calls, is the largest lever for cheaper diagnosis.
The paper's three-rule RCA (existence, correctness, consistency) could double as a checklist for building test oracles in fault-injection campaigns, giving a quantitative way to grade whether an incident report fully explains the observed error.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

SynergyRCA is a reasonable engineering contribution: an LLM-based RCA pipeline for Kubernetes that retrieves from a time-stamped StateGraph and a kind-level MetaGraph, with expert prompts and a retry loop. The graph construction is a genuinely useful idea, and the two-cluster evaluation covers many incident types. But the headline precision numbers are not reproducible as reported. The paper's own Section IV-B says snapshots are taken every five minutes without consistency guarantees, and that this measurably hurts precision (UnboundPVC at 0.26 and 0.46). It then says precision rises above 0.95 if you 'tolerate inconsistent snapshots,' without defining what that means. That one undefined step sits right between the raw results and the 0.88/0.92 average, so the central claim needs a real specification of the tolerance rule before it can be taken at face value. The stress-test note is right on this point.

What is genuinely good: the MetaGraph guides Cypher query generation so the LLM is less likely to invent impossible paths; StateChecker is explicitly told to stick to the JSON and not infer; the ReportQualityChecker retry loop is an understandable design; and the authors report token/time costs, which is more than most such papers do. They also state the snapshot limitation openly in Section V, which I credit. The related-work citations (RCACopilot, GraphRAG, K8sGPT, RCAgent) are appropriate; I don't see a citation-pattern problem.

The other weaknesses are less severe but real: no code or data release, so nothing can be independently checked; no end-to-end baseline against any prior RCA tool or even the LLM without the graph; ground truth is expert-assigned with no inter-rater agreement; precision numbers are point estimates; StateChecker is excluded from the module evaluation; and ReportQualityChecker grades conclusions produced by the same model family, which is a mild circularity concern, not fatal, and the FPR/FNR table helps.

This is not a paper with a load-bearing flaw in its architecture; it is a paper whose empirical claims are not yet fit for propagation. The right path is serious peer review with major revisions: define and justify the snapshot-tolerance rule, release artifacts or at least a reproducible subset, add an end-to-end baseline, and report uncertainty in the precision estimates. If those land, this becomes a solid reference point for AIOps and Kubernetes reliability. Anyone building an in-house RCA tool will want to read it; I wouldn't yet rely on the numbers.

Referee Report

4 major / 5 minor

Summary. The paper presents SynergyRCA, an end-to-end root cause analysis (RCA) tool for Kubernetes that combines off-the-shelf LLMs (GPT-4o) with retrieval augmentation from two graph structures: a StateGraph capturing entity states and their spatial/temporal relationships, and a MetaGraph summarizing entity-kind connections. The system first identifies the source kind from an incident message, uses an LLM-based Triage module to predict candidate root-cause kinds, retrieves metapaths from the MetaGraph, generates and executes Cypher queries over the StateGraph, and finally produces a root cause report and remediation commands through a chain of LLM modules (PathQueryGen, StateChecker, ReportGen, ReportQualityChecker). The evaluation uses data from two production Kubernetes clusters (27 nodes for one week and 88 nodes for six months), reporting average precision of 0.88 and 0.92, an average analysis time of about two minutes per attempt, and the identification of 18 and 20 root-cause types, including five types new to the second dataset. The paper also reports per-module precision metrics and token/time costs.

Significance. If the reported results hold, SynergyRCA is a practically relevant contribution: it avoids expensive fine-tuning and extensive in-context demonstrations, instead grounding LLM reasoning in structured runtime state, and it demonstrates real-world applicability on production clusters. The StateGraph/MetaGraph design is a plausible and generalizable mechanism for injecting temporal state semantics into LLM-based RCA, and the module-level evaluation provides useful insight into where such systems fail. The paper also honestly acknowledges limitations (inconsistent snapshots for fast-changing resources). However, the current evidence is weakened by the undefined 'tolerate inconsistent snapshots' adjustment, the absence of released artifacts, the lack of an end-to-end baseline, and the use of an LLM-based judge that controls the retry loop; each of these must be addressed before the central precision claims can be considered robust.

major comments (4)

[Section IV-B, Table II, and Section V] The sentence 'If we tolerate inconsistent snapshots, precision increases above 0.95' is load-bearing for the central precision claim, but the term 'tolerate' is never defined. The paper does not specify how inconsistent snapshots are detected, whether affected examples are dropped, relabeled, re-run, or corrected, or how this adjustment interacts with the per-example ground-truth labels. Because Table II shows that UnboundPVC precision is only 0.26 (dataset-1) and 0.46 (dataset-2), and the overall averages are 0.88/0.92, this undefined adjustment could materially change the headline results. Section V concedes that consistent snapshots of fast-changing resources remain an open limitation, which reinforces the need for an explicit, reproducible rule. I request a precise definition of the tolerance procedure and a re-reporting of Table II with and without the adjustment.
[Section IV-B and Table II] The evaluation protocol allows up to three trials per incident and counts an example as correct if any trial yields a report that 'reasonably explains the root cause.' Because the retry decision is made by ReportQualityChecker, an LLM from the same family as the report generator, the reported precision conflates the system's one-shot diagnostic ability with the success of the LLM-judge-controlled retry loop. The paper should report per-attempt precision (e.g., precision at trial 1, trial 2, and trial 3) or success@k, and should validate the checker's retry decisions beyond the aggregate FPR/FNR in Table III by showing how often a retry is triggered and whether retries convert initially incorrect conclusions into correct ones. Without this, the 0.88/0.92 precision numbers do not isolate the RCA quality from the self-assessment loop.
[Section IV-A and IV-B] The ground-truth construction is under-specified. The paper states that 'incident owners and senior Kubernetes administrators are invited to formulate the ground truth and evaluate the results,' but it does not state how many annotators participated, whether each example received multiple independent labels, how disagreements were resolved, or what inter-rater reliability was achieved. Given that precision is defined as the percentage of examples for which the generated report 'reasonably explains the root cause,' a subjective judgment, the absence of a documented labeling protocol leaves the precision numbers (including the per-type values in Table II) difficult to scrutinize. I ask for the annotation protocol, rater agreement, and a sample of the labeling instructions.
[Section III and IV] No code, data, or implementation artifacts are provided, and the descriptions of key construction details are insufficient for replication. For example, Section III-C says snapshots are collected 'every 5 minutes' but does not specify the exact polling mechanisms, the schema of StateJSON, the entity-extraction rules, or the thresholds and parameters used in the LLM prompts; Section IV-C relies on a linear cost model without reporting actual API latency distributions. Since the paper's contribution is a concrete system, the absence of a public artifact (even an anonymized dataset or a minimal code skeleton) is a significant reproducibility gap that should be addressed in a revision, perhaps via an artifact appendix or an external repository.

minor comments (5)

[Table III header] The header 'ReportGen-Conlusion-Precision' contains a typo; it should read 'ReportGen-Conclusion-Precision.'
[Section IV-C] The statement that omitting StateChecker means 'the overall precision (e.g. 0.88) as a lower bound for its performance' is not logically justified; the end-to-end precision does not directly bound the module-level accuracy of StateChecker, since other modules contribute errors. This should be reworded or supported with a separate analysis.
[Section IV-D and Tables IV-V] The average time and token cost are strongly influenced by the Failed ArtifactNotFound outlier in dataset-1 (741.95 seconds, 1.34M prompt tokens). Reporting medians and per-type percentiles would give a more representative efficiency picture.
[Section II and References] The name 'Ahamed' in the introduction and reference [22] is inconsistent with the spelling 'Ahmed' used in Section II-B; please standardize the citation author name.
[Section III-C] The text states that snapshots are collected 'every 5 minutes,' but the evaluation includes 'repeated 19 times' in an error message (Table I), and the time granularity of the data is not otherwise detailed; please clarify whether event logs and periodic snapshots share the same collection pipeline and how timestamps are aligned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed precision is a directly measured expert-judged ratio (Table II), with no fitted inputs and no load-bearing self-citation; the undefined 'tolerate inconsistent snapshots' aside does not enter the headline computation.

full rationale

SynergyRCA's central precision claim (0.88/0.92 in Section VI; 'approximately 0.90' in the Abstract) is a directly measured ratio of expert-judged correct reports to total examples (Total #Correct/Total #Example in Table II: 549/619 and 759/843), not the output of any fitted model or of the system's own scoring. Ground truth is formulated externally: 'Incident owners and senior Kubernetes administrators are invited to formulate the ground truth and evaluate the results' (Section IV-A). The ReportQualityChecker's LLM self-assessment only controls the bounded retry loop (max three trials, Section III-A); final correctness is decided by the same external experts, so the evaluation metric is not defined by the system's own judgment. No parameter is fitted to a subset of data and then 'predicted' on a related quantity. Building the StateGraph and MetaGraph from the same clusters used for evaluation is the intended deployment setting for a cluster-monitoring RCA tool, not a circular reduction: each diagnosis is validated by independent human experts against actual cluster state data. The only citation overlapping with the present authors is [62] (W. Xu et al., SOSP 2009), used for a 'rules and statistics-based method' for entity-key extraction (Section III-C); it is an externally published, falsifiable result whose assumptions do not include SynergyRCA's target claims, so it is real evidence and does not raise the circularity score. The sentence 'If we tolerate inconsistent snapshots, precision increases above 0.95, highlighting SynergyRCA's effectiveness using LLM' (Section IV-B) is a hypothetical aside with no defined procedure; it does not enter the Table II computation, which includes the low-precision UnboundPVC (0.26/0.46) and ExceedQuota* cases, so the headline numbers are not inflated by it. Snapshot-consistency artifacts and the unspecified 'tolerance' rule are correctness/reproducibility risks, not circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The paper is an engineering system, not a formal derivation, so the ledger records the hand-chosen settings and domain assumptions that the reported precision depends on. The main burdens are the 5-minute snapshot interval, the retry threshold policy, and the completeness assumption for the MetaGraph.

free parameters (3)

Snapshot polling interval = 5 minutes
Hand-chosen in Section III-C and IV-B; states that change faster than 5 minutes produce mismatched StateJSON and lower precision on UnboundPVC and ExceedQuota incidents.
Maximum retry trials = 3
Hand-chosen in Section IV-B ('up to three trials'); more trials would likely raise precision at higher time and token cost.
ReportQualityChecker score thresholds = 9/10, 7/10, 5/10
Hand-chosen in Fig. 10; these thresholds decide when a report is accepted and whether the system retries, directly affecting reported precision and FPR/FNR.

assumptions (4)

domain assumption Root causes can be found by checking state existence, correctness, and consistency (the state reconciliation principle).
Used in ReportGen rules (Section III-B4); the whole diagnosis rests on this modeling choice.
domain assumption The MetaGraph extracted from historical StateGraph snapshots contains every dependency path needed to diagnose new incidents.
Stated in Section III-D; if a real relationship never appeared in the snapshots, it is absent from the MetaGraph and cannot be queried.
domain assumption Ground truth labels from incident owners and senior Kubernetes administrators are correct and consistent.
Stated in Section IV-A; no inter-rater reliability is reported, so label noise is unmeasured.
domain assumption Periodic 5-minute snapshots represent the state of relevant entities at incident time.
The paper itself flags this as a limitation in Section IV-B and V; here it is the load-bearing premise for the precision numbers.

invented entities (2)

StateGraph
purpose: Graph of Kubernetes entities and their periodic state snapshots, used as the retrieval knowledge base for the LLM.
New data structure proposed by this paper; its only validation is the in-paper evaluation on two clusters, with no external benchmark or released artifact.
MetaGraph
purpose: Kind-level abstraction of the StateGraph that enumerates metapaths used to generate Cypher queries.
Derived from the StateGraph; completeness is assumed, and no independent evidence is provided beyond the paper's own precision results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM." pith.science (2026). https://pith.science/paper/5A7HCCTC

@misc{pith2026250602490,
  author       = {Pith},
  title        = {Pith review of: Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5A7HCCTC}},
  note         = {Machine review of arXiv:2506.02490}
}

read the original abstract

Kubernetes, a notably complex and distributed system, utilizes an array of controllers to uphold cluster management logic through state reconciliation. Nevertheless, maintaining state consistency presents significant challenges due to unexpected failures, network disruptions, and asynchronous issues, especially within dynamic cloud environments. These challenges result in operational disruptions and economic losses, underscoring the necessity for robust root cause analysis (RCA) to enhance Kubernetes reliability. The development of large language models (LLMs) presents a promising direction for RCA. However, existing methodologies encounter several obstacles, including the diverse and evolving nature of Kubernetes incidents, the intricate context of incidents, and the polymorphic nature of these incidents. In this paper, we introduce SynergyRCA, an innovative tool that leverages LLMs with retrieval augmentation from graph databases and enhancement with expert prompts. SynergyRCA constructs a StateGraph to capture spatial and temporal relationships and utilizes a MetaGraph to outline entity connections. Upon the occurrence of an incident, an LLM predicts the most pertinent resource, and SynergyRCA queries the MetaGraph and StateGraph to deliver context-specific insights for RCA. We evaluate SynergyRCA using datasets from two production Kubernetes clusters, highlighting its capacity to identify numerous root causes, including novel ones, with high efficiency and precision. SynergyRCA demonstrates the ability to identify root causes in an average time of about two minutes and achieves an impressive precision of approximately 0.90.

Figures

Figures reproduced from arXiv: 2506.02490 by the authors.

**Figure 2.** Prompt snippet in Triage metapath into a Cypher query. PathQueryGen enhances the metapath by prepending the sequence (“EVENT →Event →srcKind”) to it. For example, the extended metapath for issues like FailedMount-NoSuchFileDir would be: HasEvent, Event, EVENT, metadata_uid; ReferInternal, Event, Pod, involvedObject_uid; ReferInternal, Pod, PersistentVolumeClaim, spec_volumes_persistentVolumeClaim_claimName; ReferInt… view at source ↗

**Figure 5.** An example statepath of FailedCreateExceedQuotaJob with Event and Snapshots extended. intricate structure and data types present in States. This complexity is further complicated by varying keys across different entity kinds, and even within a single kind. StateChecker excels in extracting the most pertinent information, providing significant advantages in navigating these complexities. We execute the generated Cy… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Prompt snippet in PathQueryGen [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 8.** Figure 8: Prompt snippet in ReportGen 1 {"summary": [ 2 {"kind": "Job", 3 "explanation": "The Job object specifies creating a 4 new pod in the 'foobar3' namespace, which is at 5 its pod quota limit.", 6 "relevance_score": "9/10" 7 }, 8 {"kind": "Namespace", 9 "explanation": "The…

**Figure 6.** Figure 6: Prompt snippet in StateChecker 1 RESOURCEQUOTA(25f8e934-6e18-5b9c-bee4-b4231a9d01a6): 2 Key Observations: 1. The error message indicates exceeding the pod quota in the ‘foobar3’ namespace. 3 4 5 2. The JSON object is a ResourceQuota for the ‘foobar3’ namespace. 3. The…

**Figure 9.** Figure 9: An example RCA report and recommended solution [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]

**Figure 10.** Figure 10: Prompt snippet in ReportQualityChecker C. Constructing StateGraph from Running Kubernetes Cluster This section introduces StateGraph, a graph structure designed to capture the spatial and temporal relationships among entities within a Kubernetes environment. StateGra…

**Figure 12.** Figure 12: An example StateGraph snippet for a single Pod [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 52 canonical work pages

[1]

Kubernetes:production-grade container orchestration,

K8s-project, “Kubernetes:production-grade container orchestration,” https://kubernetes.io/, 2024, accessed: 2024-11-24

work page 2024
[2]

Large-scale cluster management at google with borg,

A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the tenth european conference on computer systems, 2015, pp. 1–17

work page 2015
[3]

Elastic cloud services: scaling snowflake’s control plane,

T. Melissaris, K. Nabar, R. Radut, S. Rehmtulla, A. Shi, S. Chan- drashekar, and I. Papapanagiotou, “Elastic cloud services: scaling snowflake’s control plane,” inProceedings of the 13th Symposium on Cloud Computing, 2022, pp. 142–157

work page 2022
[4]

Twine: A unified cluster management system for shared infrastructure,

C. Tang, K. Yu, K. Veeraraghavan, J. Kaldor, S. Michelson, T. Kooburat, A. Anbudurai, M. Clark, K. Gogia, L. Chenget al., “Twine: A unified cluster management system for shared infrastructure,” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 787–803

work page 2020
[5]

Cloud native 2023: The undisputed infrastructure of global tech- nology,

CNCF, “Cloud native 2023: The undisputed infrastructure of global tech- nology,” https://www.cncf.io/reports/cncf-annual-survey-2023/, 2023, accessed: 2024-11-24

work page 2023
[6]

Controllers and reconciliation,

K8s-project, “Controllers and reconciliation,” https://cluster-api. sigs.k8s.io/developer/providers/implementers-guide/controllers and reconciliation.html, 2024, accessed: 2024-11-24

work page 2024
[7]

Borg, omega, and kubernetes,

B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, omega, and kubernetes,”Communications of the ACM, vol. 59, no. 5, pp. 50–57, 2016

work page 2016
[8]

Acto: Automatic end-to-end testing for operation correctness of cloud system management,

J. T. Gu, X. Sun, W. Zhang, Y . Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu, “Acto: Automatic end-to-end testing for operation correctness of cloud system management,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 96–112

work page 2023

Show all 69 references

[9]

Anvil: Verifying liveness of cluster management controllers,

X. Sun, W. Ma, J. T. Gu, Z. Ma, T. Chajed, J. Howell, A. Lattuada, O. Padon, L. Suresh, A. Szekereset al., “Anvil: Verifying liveness of cluster management controllers,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 649–666

2024
[10]

Automatic reliability testing for cluster manage- ment controllers,

X. Sun, W. Luo, J. T. Gu, A. Ganesan, R. Alagappan, M. Gasch, L. Suresh, and T. Xu, “Automatic reliability testing for cluster manage- ment controllers,” in16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 143–159

2022
[11]

On byzantine fault tolerance in multi-master kubernetes clusters,

G. M. Diouf, H. Elbiaze, and W. Jaafar, “On byzantine fault tolerance in multi-master kubernetes clusters,”Future Generation Computer Systems, vol. 109, pp. 407–419, 2020

2020
[12]

Kubernetes failure stories,

H. Jacobs, “Kubernetes failure stories,” https://k8s.af/, 2024, accessed: 2024-11-24

2024
[13]

Weird ways to blow up your kubernetes,

M. Cebula and B. Sherrod, “Weird ways to blow up your kubernetes,” KubeCon North America (Nov. 2019), 10

2019
[14]

Preventing controller sprawl from taking down your cluster,

C. Madhu, “Preventing controller sprawl from taking down your cluster,” KubeCon North America (Oct. 2022), 2022

2022
[15]

Zookeeper pod keeps crashing when scaling down and up,

Github, “Zookeeper pod keeps crashing when scaling down and up,” https://github.com/pravega/zookeeper-operator/pull/526, 2024, accessed: 2024-11-24

2024
[16]

[bug] tidb operator unable to recover an unhealthy cluster even with manual revert,

CASSKOP-370., “[bug] tidb operator unable to recover an unhealthy cluster even with manual revert,” https://github.com/ Orange-OpenSource/casskop/issues/370, 2024, accessed: 2024-11-24

2024
[17]

Tidb operator unable to recover an unhealthy cluster even with manual revert,

Github, “Tidb operator unable to recover an unhealthy cluster even with manual revert,” https://github.com/pingcap/tidb-operator/issues/ 4946, 2024, accessed: 2024-11-24

2024
[18]

Jigsaw: Large language models meet program synthesis,

N. Jain, S. Vaidyanath, A. Iyer, N. Natarajan, S. Parthasarathy, S. Ra- jamani, and R. Sharma, “Jigsaw: Large language models meet program synthesis,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 1219–1231

2022
[19]

Using deep learning to generate complete log statements,

A. Mastropaolo, L. Pascarella, and G. Bavota, “Using deep learning to generate complete log statements,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 2279– 2290

2022
[20]

Improving automat- ically generated code from codex via automated program repair,

Z. Fan, X. Gao, A. Roychoudhury, and S. H. Tan, “Improving automat- ically generated code from codex via automated program repair,”arXiv preprint arXiv:2205.10583, 2022

2022 arXiv
[21]

Studying the usage of text-to-text transfer transformer to support code-related tasks,

A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshy- vanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 202...

2021
[22]

Recommending root-cause and mitigation steps for cloud incidents using large language models,

T. Ahmed, S. Ghosh, C. Bansal, T. Zimmermann, X. Zhang, and S. Rajmohan, “Recommending root-cause and mitigation steps for cloud incidents using large language models,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1737–1749

2023
[23]

Automated root causing of cloud incidents using in-context learning with gpt-4,

X. Zhang, S. Ghosh, C. Bansal, R. Wang, M. Ma, Y . Kang, and S. Ra- jmohan, “Automated root causing of cloud incidents using in-context learning with gpt-4,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 266–277

2024
[24]

Automatic root cause analysis via large language models for cloud incidents,

Y . Chen, H. Xie, M. Ma, Y . Kang, X. Gao, L. Shi, Y . Cao, X. Gao, H. Fan, M. Wenet al., “Automatic root cause analysis via large language models for cloud incidents,” inProceedings of the Nineteenth European Conference on Computer Systems, 2024, pp. 674–688

2024
[25]

k8sgpt: Giving kubernetes superpowers to everyone,

K8sGPT-project, “k8sgpt: Giving kubernetes superpowers to everyone,” https://github.com/k8sgpt-ai/k8sgpt, 2024, accessed: 2024-11-24

2024
[26]

X-lifecycle learning for cloud incident management using llms,

D. Goel, F. Husain, A. Singh, S. Ghosh, A. Parayil, C. Bansal, X. Zhang, and S. Rajmohan, “X-lifecycle learning for cloud incident management using llms,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 417– 428

2024
[27]

Neo4j graph database and analytics,

Neo4j, “Neo4j graph database and analytics,” https://neo4j.com/, 2024, accessed: 2024-11-27

2024
[28]

Picking pearl from seabed: Extracting arte- facts from noisy issue triaging collaborative conversations for hybrid cloud services,

A. P. Azad, S. Ghosh, A. Gupta, H. Kumar, P. Mohapatra, L. Eckstein, L. Posner, and R. Kern, “Picking pearl from seabed: Extracting arte- facts from noisy issue triaging collaborative conversations for hybrid cloud services,” inProceedings of the AAAI Conference on Artificial ...

2022
[29]

An empirical investigation of incident triage for online service systems,

J. Chen, X. He, Q. Lin, Y . Xu, H. Zhang, D. Hao, F. Gao, Z. Xu, Y . Dang, and D. Zhang, “An empirical investigation of incident triage for online service systems,” in2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SE...

2019
[30]

Continuous incident triage for large-scale online service sys- tems,

J. Chen, X. He, Q. Lin, H. Zhang, D. Hao, F. Gao, Z. Xu, Y . Dang, and D. Zhang, “Continuous incident triage for large-scale online service sys- tems,” in2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 364–375

2019
[31]

Identifying linked incidents in large-scale online service systems,

Y . Chen, X. Yang, H. Dong, X. He, H. Zhang, Q. Lin, J. Chen, P. Zhao, Y . Kang, F. Gaoet al., “Identifying linked incidents in large-scale online service systems,” inProceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the fou...

2020
[32]

Dependency aware incident linking in large cloud systems,

S. Ghosh, K. Grover, J. Wong, C. Bansal, R. Namineni, M. Verma, and S. Rajmohan, “Dependency aware incident linking in large cloud systems,” inCompanion Proceedings of the ACM on Web Conference 2024, 2024, pp. 141–150

2024
[33]

Decaf: Diagnosing and triaging performance issues in large-scale cloud services,

C. Bansal, S. Renganathan, A. Asudani, O. Midy, and M. Janakiraman, “Decaf: Diagnosing and triaging performance issues in large-scale cloud services,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, 2020, p...

2020
[34]

Correlating events with time series for incident diagnosis,

C. Luo, J.-G. Lou, Q. Lin, Q. Fu, R. Ding, D. Zhang, and Z. Wang, “Correlating events with time series for incident diagnosis,” inProceed- ings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014, pp. 1583–1592

2014
[35]

Learning a hierarchical monitoring system for detecting and diagnosing service issues,

V . Nair, A. Raul, S. Khanduja, V . Bahirwani, Q. Shao, S. Sellamanickam, S. Keerthi, S. Herbert, and S. Dhulipalla, “Learning a hierarchical monitoring system for detecting and diagnosing service issues,” in Proceedings of the 21th ACM SIGKDD international conference on knowl...

2015
[36]

How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems,

J. Jiang, W. Lu, J. Chen, Q. Lin, P. Zhao, Y . Kang, H. Zhang, Y . Xiong, F. Gao, Z. Xuet al., “How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems,” inProceedings of the 28th ACM Joint Meeting on European Softwa...

2020
[37]

Log clustering based problem identification for online service systems,

Q. Lin, H. Zhang, J.-G. Lou, Y . Zhang, and X. Chen, “Log clustering based problem identification for online service systems,” inProceedings of the 38th International Conference on Software Engineering Compan- ion, 2016, pp. 102–111

2016
[38]

An approach to cloud execution failure diagnosis based on exception logs in openstack,

Y . Yuan, W. Shi, B. Liang, and B. Qin, “An approach to cloud execution failure diagnosis based on exception logs in openstack,” in2019 IEEE 12th International Conference on Cloud Computing (CLOUD). IEEE, 2019, pp. 124–131

2019
[39]

Eadro: An end- to-end troubleshooting framework for microservices on multi-source data,

C. Lee, T. Yang, Z. Chen, Y . Su, and M. R. Lyu, “Eadro: An end- to-end troubleshooting framework for microservices on multi-source data,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1750–1762

2023
[40]

Robust failure diagnosis of microservice system through multimodal data,

S. Zhang, P. Jin, Z. Lin, Y . Sun, B. Zhang, S. Xia, Z. Li, Z. Zhong, M. Ma, W. Jinet al., “Robust failure diagnosis of microservice system through multimodal data,”IEEE Transactions on Services Computing, vol. 16, no. 6, pp. 3851–3864, 2023

2023
[41]

Root cause analysis of failures in microservices through causal discovery,

A. Ikram, S. Chakraborty, S. Mitra, S. Saini, S. Bagchi, and M. Ko- caoglu, “Root cause analysis of failures in microservices through causal discovery,”Advances in Neural Information Processing Systems, vol. 35, pp. 31 158–31 170, 2022

2022
[42]

Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback,

L. Wang, C. Zhang, R. Ding, Y . Xu, Q. Chen, W. Zou, Q. Chen, M. Zhang, X. Gao, H. Fanet al., “Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and ...

2023
[43]

Causal inference-based root cause analysis for online service systems with intervention recognition,

M. Li, Z. Li, K. Yin, X. Nie, W. Zhang, K. Sui, and D. Pei, “Causal inference-based root cause analysis for online service systems with intervention recognition,” inProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 3230– 3240

2022
[44]

Practical root cause localization for microservice systems via trace analysis,

Z. Li, J. Chen, R. Jiao, N. Zhao, Z. Wang, S. Zhang, Y . Wu, L. Jiang, L. Yan, Z. Wanget al., “Practical root cause localization for microservice systems via trace analysis,” in2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS). IEEE, 2021, pp. 1–10

2021
[45]

Mutiny! how does kubernetes fail, and what can we do about it?

M. Barletta, M. Cinque, C. Di Martino, Z. T. Kalbarczyk, and R. K. Iyer, “Mutiny! how does kubernetes fail, and what can we do about it?” arXiv preprint arXiv:2404.11169, 2024

2024 arXiv
[46]

Iot anomaly detection via device interaction graph,

J. Wang, Z. Li, M. Sun, B. Yuan, and J. C. Lui, “Iot anomaly detection via device interaction graph,” in2023 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 2023, pp. 494–507

2023
[47]

Rapminer: A generic anomaly localization mechanism for cdn system with multi-dimensional kpis,

C. Liu, Y . Liu, Z. Xu, and L. Dai, “Rapminer: A generic anomaly localization mechanism for cdn system with multi-dimensional kpis,” in 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 2022, pp. 319–330

2022
[48]

A survey of aiops for failure management in the era of large language models,

L. Zhang, T. Jia, M. Jia, Y . Yang, Z. Wu, and Y . Li, “A survey of aiops for failure management in the era of large language models,”arXiv preprint arXiv:2406.11213, 2024

2024 arXiv
[49]

Graph-based root cause analysis for service-oriented and microservice architectures,

´A. Brand ´on, M. Sol ´e, A. Hu ´elamo, D. Solans, M. S. P ´erez, and V . Munt´es-Mulero, “Graph-based root cause analysis for service-oriented and microservice architectures,”Journal of Systems and Software, vol. 159, p. 110432, 2020

2020
[50]

Microrca: Root cause localization of performance issues in microservices,

L. Wu, J. Tordsson, E. Elmroth, and O. Kao, “Microrca: Root cause localization of performance issues in microservices,” inNOMS 2020- 2020 IEEE/IFIP Network Operations and Management Symposium. IEEE, 2020, pp. 1–9

2020
[51]

Performance diagnosis in cloud microservices using deep learning,

L. Wu, J. Bogatinovski, S. Nedelkoski, J. Tordsson, and O. Kao, “Performance diagnosis in cloud microservices using deep learning,” inInternational Conference on Service-Oriented Computing. Springer, 2020, pp. 85–96

2020
[52]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[53]

A systematic evaluation of large language models of code,

F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” inProceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10

2022
[54]

Repair is nearly generation: Multilingual program repair with llms,

H. Joshi, J. C. Sanchez, S. Gulwani, V . Le, G. Verbruggen, and I. Radi ˇcek, “Repair is nearly generation: Multilingual program repair with llms,” inProceedings of the AAAI Conference on Artificial Intelli- gence, vol. 37, no. 4, 2023, pp. 5131–5140

2023
[55]

Vulre- pair: a t5-based automated software vulnerability repair,

M. Fu, C. Tantithamthavorn, T. Le, V . Nguyen, and D. Phung, “Vulre- pair: a t5-based automated software vulnerability repair,” inProceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 935– 947

2022
[56]

Rcagent: Cloud root cause analysis by autonomous agents with tool- augmented large language models,

Z. Wang, Z. Liu, Y . Zhang, A. Zhong, L. Fan, L. Wu, and Q. Wen, “Rcagent: Cloud root cause analysis by autonomous agents with tool- augmented large language models,”arXiv preprint arXiv:2310.16340, 2023

2023 arXiv
[57]

Face it yourselves: An llm-based two-stage strategy to localize configuration errors via logs,

S. Shan, Y . Huo, Y . Su, Y . Li, D. Li, and Z. Zheng, “Face it yourselves: An llm-based two-stage strategy to localize configuration errors via logs,”arXiv preprint arXiv:2404.00640, 2024

2024 arXiv
[58]

Exploring llm-based agents for root cause analysis,

D. Roy, X. Zhang, R. Bhave, C. Bansal, P. Las-Casas, R. Fonseca, and S. Rajmohan, “Exploring llm-based agents for root cause analysis,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 208–219

2024
[59]

Welcome to graphrag,

Microsoft, “Welcome to graphrag,” https://microsoft.github.io/graphrag/, 2024, accessed: 2024-11-24

2024
[60]

Cypher manual for neo4j graph data platform,

Cypher, “Cypher manual for neo4j graph data platform,” https://neo4j. com/docs/cypher-manual/current/introduction/, 2024, accessed: 2024- 11-27

2024
[61]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

2022
[62]

Largescale system problem detection by mining console logs,

W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, “Largescale system problem detection by mining console logs,” inProceedings of SOSP, vol. 9. Citeseer, 2009, pp. 1–17

2009
[63]

Spark: Cluster computing with working sets,

M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster computing with working sets,” in2nd USENIX workshop on hot topics in cloud computing (HotCloud 10), 2010

2010
[64]

Unified engine for large-scale data analytics,

Spark, “Unified engine for large-scale data analytics,” https://spark. apache.org/, 2024, accessed: 2024-11-24

2024
[65]

Graphframes: an integrated api for mixing graph and relational queries,

A. Dave, A. Jindal, L. E. Li, R. Xin, J. Gonzalez, and M. Zaharia, “Graphframes: an integrated api for mixing graph and relational queries,” inProceedings of the fourth international workshop on graph data management experiences and systems, 2016, pp. 1–8

2016
[66]

Graphframes overview,

Spark, “Graphframes overview,” https://graphframes.github.io/ graphframes/docs/ site/index.html, 2024, accessed: 2024-11-24

2024
[67]

Assistants api overview beta,

OpenAI, “Assistants api overview beta,” https://platform.openai.com/ docs/assistants/overview, 2024, accessed: 2024-11-24

2024
[68]

Quickstart: Get started using azure openai assistants (preview),

Azure, “Quickstart: Get started using azure openai assistants (preview),” https://learn.microsoft.com/en-us/azure/ai-services/openai/ assistants-quickstart?tabs=command-line%2Ctypescript&pivots= programming-language-python, 2024, accessed: 2024-11-24

2024
[69]

Pricing: Simple and flexible. only pay for what you use

OpenAI, “Pricing: Simple and flexible. only pay for what you use.” https://openai.com/api/pricing/, 2024, accessed: 2024-11-24

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Kubernetes:production-grade container orchestration,

K8s-project, “Kubernetes:production-grade container orchestration,” https://kubernetes.io/, 2024, accessed: 2024-11-24

work page 2024

[2] [2]

Large-scale cluster management at google with borg,

A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the tenth european conference on computer systems, 2015, pp. 1–17

work page 2015

[3] [3]

Elastic cloud services: scaling snowflake’s control plane,

T. Melissaris, K. Nabar, R. Radut, S. Rehmtulla, A. Shi, S. Chan- drashekar, and I. Papapanagiotou, “Elastic cloud services: scaling snowflake’s control plane,” inProceedings of the 13th Symposium on Cloud Computing, 2022, pp. 142–157

work page 2022

[4] [4]

Twine: A unified cluster management system for shared infrastructure,

C. Tang, K. Yu, K. Veeraraghavan, J. Kaldor, S. Michelson, T. Kooburat, A. Anbudurai, M. Clark, K. Gogia, L. Chenget al., “Twine: A unified cluster management system for shared infrastructure,” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 787–803

work page 2020

[5] [5]

Cloud native 2023: The undisputed infrastructure of global tech- nology,

CNCF, “Cloud native 2023: The undisputed infrastructure of global tech- nology,” https://www.cncf.io/reports/cncf-annual-survey-2023/, 2023, accessed: 2024-11-24

work page 2023

[6] [6]

Controllers and reconciliation,

K8s-project, “Controllers and reconciliation,” https://cluster-api. sigs.k8s.io/developer/providers/implementers-guide/controllers and reconciliation.html, 2024, accessed: 2024-11-24

work page 2024

[7] [7]

Borg, omega, and kubernetes,

B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, omega, and kubernetes,”Communications of the ACM, vol. 59, no. 5, pp. 50–57, 2016

work page 2016

[8] [8]

Acto: Automatic end-to-end testing for operation correctness of cloud system management,

J. T. Gu, X. Sun, W. Zhang, Y . Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu, “Acto: Automatic end-to-end testing for operation correctness of cloud system management,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 96–112

work page 2023