Pith. sign in

REVIEW 4 major objections 4 minor 66 references

ChaosEater: Fully Automating Chaos Engineering with Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pipeline of LLM agents can complete a full chaos-engineering cycle on Kubernetes systems autonomously, finding and fixing fragility in minutes for under a dollar.

desk verdict A well-built, fully open proof of concept for LLM-driven chaos engineering, but the central claim rests on a thin, partly self-referential evaluation. read the letter →

arxiv 2501.11107 v2 pith:A4MVPADE submitted 2025-01-19 cs.SE cs.AIcs.CLcs.DCcs.NI

Daisuke Kikuta , Hiroki Ikeuchi , Kengo Tajiri This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.DCcs.NI
keywords chaosengineeringKuberneteslargelanguagemodelsLLMagentsinfrastructureascodefaultinjectionresiliencytestingValidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CHAOS EATER, a system that uses a fixed workflow of LLM agents to carry out an entire chaos-engineering cycle on Kubernetes systems managed as code: it defines steady states and failure scenarios, plans and executes fault injection, analyzes whether the system stayed healthy, and reconfigures the Kubernetes manifests when it did not. The central claim is that this full loop completes stably and cheaply, at roughly $0.21 to $0.84 in API cost and 11 to 25 minutes per cycle in the two case studies, and that human engineers and three LLM judges rate the resulting cycles as reasonable. If true, the work matters because it turns resilience testing from a manual, expert-only activity into an automated software-engineering task.

What carries the argument

The carrying object is a predefined agentic workflow of twenty LLM agents chained through the phases pre-processing, hypothesis, experiment, analysis, improvement, and post-processing, with rule-based verification loops at script-generation and manifest-application points. The load-bearing idea inside it is Validation as Code: each steady state is a pair of a measurable state and a threshold, and the threshold is turned into a unit-test script (Python against the Kubernetes API, or k6 JavaScript for communication metrics) whose assertions decide whether the hypothesis held. These scripts and the chosen failure injections are compiled by a hierarchical node-grouping algorithm into a Chaos Mesh workflow manifest, which schedules pre-validation, failure-injection, and post-validation stages automatically. That conversion is what lets the LLM define intent while the execution and judgment remain deterministic.

What would settle it

Run CHAOS EATER on a Kubernetes system with a planted, non-obvious vulnerability, such as a single replica whose readiness probe fails only under partial network partition, with no hint about the defect, and check whether the generated VaC thresholds and failure injections detect it. If the cycle completes without reconfiguration because all LLM-chosen thresholds are satisfied by construction, the claim that the pipeline performs meaningful resilience testing is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the entire systematic chaos-engineering cycle can be delegated to LLMs by fixing the workflow and subdividing operations among twenty role-specialized agents, and that this delegation is reliable enough to complete single cycles on both a two-manifest NGINX system and the 29-manifest Sock Shop system without runtime errors, reconfiguring the system appropriately in five of five and four of five runs respectively. The key mechanism for trustworthy validation is 'Validation as Code' (VaC): steady states are encoded as executable unit tests with threshold assertions, so the experiment's pass/fail judgment is made by code rather than by an LLM reading logs. The paper validates the cycles qualitatively with two external human engineers and three LLMs, all of whom rated every phase at or above the positive threshold.

Load-bearing premise

The result depends on the LLM-chosen steady-state thresholds and failure scenario being a real test of the system; the threshold agent is told to set thresholds that the current, healthy state already satisfies, so a cycle can pass or yield only trivial fixes if the LLM picks an easy target.

Editorial extensions

If this is right

  • A single operator can hand a folder of Kubernetes manifests to CHAOS EATER and receive, in under half an hour, a summary plus a hardened manifest set, replacing the manual hypothesis-definition, experiment-planning, and reconfiguration work.
  • Cost grows slowly with system size: roughly quadrupling API cost and doubling time when moving from a two-manifest NGINX system to the 29-manifest Sock Shop, suggesting the approach is feasible beyond toy deployments.
  • Because steady-state validation is encoded as unit tests, the same judgment is reproducible run-to-run; the LLM proposes, and the code disposes.
  • The reconfiguration actions the system takes are standard resilience fixes, such as replacing a bare Pod with a Deployment and raising replica counts, so the pipeline can implement a meaningful subset of resiliency improvements without human intervention.
  • Qualitative agreement between human engineers and LLM judges on all phases suggests the cycle output is understandable and auditable by people.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit is a hidden-vulnerability benchmark: plant a defect that is not detectable by manifest inspection alone and check whether the self-chosen thresholds and failures actually expose it; without such a benchmark, a completed cycle could be vacuous.
  • The paper itself notes that on already-resilient systems the current system does not find hidden issues; this suggests the pipeline's present value is catching simple configuration fragility, and that long-term multiple cycles need history management and continuous learning to reach deeper fault discovery.
  • The same workflow design could be ported to other code-defined infrastructure, such as Terraform or Helm, or to non-Kubernetes targets, by swapping the manifest parser and the fault-injection backend.
  • Automatic prompt tuning and workflow optimization, which the paper identifies as future work, could make the system model-agnostic and reduce the manual prompt-management cost of the twenty-agent design.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. ChaosEater is an LLM-based agentic system that aims to automate the full chaos engineering (CE) cycle for Kubernetes systems. Given a folder of K8s manifests and a Skaffold configuration, it performs preprocessing, hypothesis definition (steady states plus a failure scenario), experiment planning and execution via Chaos Mesh, analysis of failed validations, and reconfiguration of K8s manifests, repeating the experiment/analysis/improvement loop until the hypothesis is satisfied. Steady states are encoded as unit-test scripts ("Validation as Code"). The evaluation consists of two case studies, NGINX and Sock Shop, with five runs each; the paper reports time and API costs, plus qualitative ratings of one selected run per system by two human engineers and three LLM judges. The central claim is that ChaosEater "stably completes reasonable single CE cycles" at low cost.

Significance. If the central claim holds, this is a meaningful step toward end-to-end automation of chaos engineering: it removes manual hypothesis definition, experiment planning, failure analysis, and K8s-manifest reconfiguration, and it does so with an open-source implementation, published prompt templates, and a reproducible temperature-0/seed-42 setup. The Validation-as-Code idea, in particular, is a useful contribution because it makes steady-state checking explicit, transparent, and machine-executable. The significance is currently bounded by evaluation validity: the evidence consists of five runs on two small systems with deliberately planted, easily detectable faults, the steady-state thresholds are generated by the same LLM family that later judges the results, and the paper itself concedes in Appendix B.4 that the system fails to find hidden issues in already-resilient systems. The strength of the paper is its architecture and open artifact, not yet the demonstrated generality of the behavioral claim.

major comments (4)
  1. [Section 2.2 (Agent #1-2 and Agent #1-5 prompts)] The steady states and failure injections that define a 'reasonable CE cycle' are generated by the same LLM pipeline that is later judged, and the prompts explicitly instruct the threshold agent to make thresholds 'more easily satisfied' and to ensure the current value satisfies the threshold. This anchors pass/fail criteria to normal operation rather than to an externally defined SLO. The Sock Shop example in Figure 2 illustrates the risk: the chosen steady states are ready-replica counts, while the Black Friday scenario includes StressChaos on carts-db; CPU/memory stress can degrade latency and user-facing availability without changing ready-replica counts, so the VaC scripts can pass while the service is substantially degraded. To support the abstract's claim, the paper needs an independent validation that the generated thresholds and failure scenarios constitute a meaningful resilience test, for example a human-audited SLO benchmark or a comparison against pre-specified thresholds.
  2. [Section 3, 'Costs and stability'] The stability claim rests on five runs per system, and one of the five Sock Shop runs did not perform the expected front-end reconfiguration. The paper states that a valid cycle was completed without reconfiguration, but it does not report what steady states, failure scenario, and VaC results that run produced, nor does it define what counts as a 'successful' cycle when no issue is found. With n=5 and one divergent run, the statement 'stably completes reasonable single CE cycles' is not established. The authors should report per-run variation, define success criteria independent of whether reconfiguration happens, and ideally increase the number of runs or use multiple seeds.
  3. [Appendix B.4 and Conclusion] The paper's own limitation statement says that for systems with existing resilience, ChaosEater 'fails to find new hidden issues through a CE cycle.' This directly bounds the central claim: the two case-study systems were configured with simple, planted faults (restartPolicy: Never and a single front-end replica), so the successful runs demonstrate execution on easy cases, not general ability to discover meaningful vulnerabilities. The abstract and conclusion should either scope the claim to systems with known simple resiliency issues or provide evidence on systems where the issues are not pre-planted and obvious.
  4. [Section 3 and Appendix C.3, qualitative validation] The qualitative validation uses a self-authored rubric, only one selected run per system, two external human engineers, and three LLM judges of which one is GPT-4o, the same model family used to run ChaosEater. All evaluators rated every phase above the positive threshold, but no inter-rater reliability or disagreement analysis is reported, and the risk of circularity from having the same model family both generate and judge the cycle is not addressed. The authors should report per-evaluator scores, a sensitivity analysis excluding the GPT-4o judge, and a clearer justification for why the selected runs are representative of the five-run set.
minor comments (4)
  1. [Section 2.2] The text says the workflow goes through 'five divided phases' but then lists six phase names: pre-processing, hypothesis, experiment, analysis, improvement, and post-processing. Please correct the count or the list.
  2. [Table 1] The column header 'NGNIX' is a typo and should read 'NGINX'.
  3. [Appendix C.3] The model identifier 'gpt-4o-2024-0908-06' appears inconsistent with 'gpt-4o-2024-08-06' used in Section 3; please unify the naming.
  4. [Table 2, Experiment score 1] The criterion for a score of 1 says 'The experiment plan does serve to validate the hypothesis at all'; this should read 'does not serve'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim rests on measured cost/stability data and external human validation, not on a self-derived prediction.

full rationale

ChaosEater is an empirical systems paper rather than a formal derivation. The reported costs ($0.21–$0.84) and runtimes (11–25m) are measured directly from API token usage and execution logs, not predicted from fitted parameters. The pass/fail outcome of each cycle is determined by VaC scripts executed against a live Kind cluster during real Chaos Mesh fault injection; the threshold is fixed before the experiment and the failure is actually injected, so success is not forced by construction. Although the threshold agent is instructed to choose thresholds that are satisfied under the current state and to 'include reasonable tolerance that makes the threshold being more easily satisfied,' this is the standard CE definition of a steady state (normal behavior), and the injected failures can and did violate the thresholds (e.g., the NGINX pod-kill failure produced a failed VaC script). The main self-referential element is that GPT-4o, the model family used to generate the cycles, is also one of three LLM judges; however, the 'reasonable cycle' claim also rests on two external human engineers from different IT companies and on two other LLM families, so the conclusion does not reduce to self-evaluation. Appendix B.4's admission that ChaosEater 'fails to find new hidden issues' on already-resilient systems is an honest scope limitation about vulnerability discovery, not evidence that the claimed cycles are equivalent to their inputs. No load-bearing self-citation or imported uniqueness theorem appears.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new physical entities or fitted numerical parameters are introduced. The system's thresholds are generated by LLMs rather than fit by the authors, and Validation as Code is a method, not an invented entity.

assumptions (4)
  • domain assumption The four-phase chaos engineering cycle from Basiri et al. (hypothesis, experiment, analysis, improvement) is a valid systematic method for improving system resiliency.
    The entire workflow and evaluation rubric are built on this cycle (Section 1, Appendix C.3).
  • domain assumption Kubernetes system resiliency can be improved by editing only K8s manifest files, with no other code changes.
    The system only reconfigures K8s manifests; the authors list this as a limitation in Appendix B.
  • domain assumption Metric outputs from the K8s API and k6 are sufficient and accurate measures of the defined steady states.
    VaC scripts use these tools as ground truth for threshold checks (Section 2.2, Appendix D.5.2).
  • domain assumption GPT-4o with temperature 0 and seed 42 behaves consistently enough to support the stability claims.
    The paper itself notes different behaviors were observed despite the fixed seed and temperature (Appendix D.4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChaosEater: Fully Automating Chaos Engineering with Large Language Models." pith.science (2026). https://pith.science/paper/A4MVPADE

@misc{pith2026250111107,
  author       = {Pith},
  title        = {Pith review of: ChaosEater: Fully Automating Chaos Engineering with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4MVPADE}},
  note         = {Machine review of arXiv:2501.11107}
}
read the original abstract

Chaos Engineering (CE) is an engineering technique aimed at improving the resiliency of distributed systems. It involves artificially injecting specific failures into a distributed system and observing its behavior in response. Based on the observation, the system can be proactively improved to handle those failures. Recent CE tools implement the automated execution of predefined CE experiments. However, defining these experiments and improving the system based on the experimental results still remain manual. To reduce the costs of the manual operations, we propose ChaosEater, a system for automating the entire CE operations with Large Language Models (LLMs). It predefines the agentic workflow according to a systematic CE cycle and assigns subdivided operations within the workflow to LLMs. ChaosEater targets CE for Kubernetes systems, which are managed through code (i.e., Infrastructure as Code). Therefore, the LLMs in ChaosEater perform software engineering tasks to complete CE cycles, including requirement definition, code generation, debugging, and testing. We evaluate ChaosEater through case studies on both small and large Kubernetes systems. The results demonstrate that it stably completes reasonable single CE cycles with significantly low time and monetary costs. The CE cycles are also qualitatively validated by human engineers and LLMs.

Figures

Figures reproduced from arXiv: 2501.11107 by the authors.

Figure 1
Figure 1. A simplified agentic workflow of CHAOSEATER. CHAOSEATER follows the workflow to autonomously complete the systematic CE cycle using LLM agents and existing tools. Note that only the representative inputs and outputs of agents are illustrated here. The two K8s clusters within the workflow refer to the same one. workflow design from input to output, breaking it down into the five phases. Note that, in this paper, we r… view at source ↗
Figure 2
Figure 2. The highlighted outputs for NGINX and SOCKSHOP. See Appendix E.1 and E.3 for their full versions. completes the CE cycle for each system without runtime errors in all five runs. It also correctly reconfigures NGINX in all five runs and SOCK￾SHOP in four out of five runs. Even in the non￾reconfigured case, we confirm that a valid CE cycle is completed without requiring reconfigurations. Qualitative validation To vali… view at source ↗
Figure 3
Figure 3. Qualitative evaluation results of CE cycles for each system. A score of 3 or higher is a positive rating. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The deployment environment of CHAOSEATER. D Implementation Details D.1 System Deployment [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: The GUI of CHAOSEATER erations for the verification loop and improvement loop. If the loop exceeds this limit, an assertion error will occur, immediately terminating the app at that point. (d) Token usage You can monitor token usage in real-time. The total cost is calc…
Figure 6
Figure 6. Figure 6: The agentic workflow of CHAOSEATER vantages of dividing operations into smaller tasks and organizing the agentic workflow as shown in [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Examples of unit-test scripts to validate steady states. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: An example of detailed parameters. ing StressChaos and NetworkChaos. In step 2, the agent separately defines the de￾tailed parameters of each failure. Each failure type requires a different parameter set. Therefore, given a failure type name, CHAOSEATER dynamically sel…
Figure 9
Figure 9. Figure 9: YAML code snippets for each node type. The blue double curly braces [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Hierarchical grouping for implementing a complex chaos experiment plan in Chaos Mesh. [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Reconfiguration process by the agent and a [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: An example of prefills. D.5.1 Pre-processing Prompt 2: Agent # 0-0 for summarizing K8s manifests System: System: You are a professional Kubernetes ( k8s) engineer. Given a K8s manifest, please summarize it according to the following rules: 31 [PITH_FULL_IMAGE:figures…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 61 canonical work pages

  1. [1]

    Limited deployment environment ; Although CE should ideally be con- ducted in actual production environments, CHAOS EATER is currently only supported in development environments

    work page

  2. [2]

    Therefore, other LLMs can not cur- rently be used for CHAOS EATER

    Limited to GPT-4o ; CHAOS EATER’s prompt templates are highly tuned only for GPT-4o. Therefore, other LLMs can not cur- rently be used for CHAOS EATER

    work page

  3. [3]

    Limited to K8s manifest reconfiguration; Software systems consist not only of K8s manifests but also of other types of code- bases, such as HTML/CSS/JS and Python. Although K8s manifest reconfiguration can handle a majority of system resiliency issues, reconfiguration of all types of codebases is necessary to optimally improve system re- siliency. However...

    work page

  4. [4]

    However, for systems that already possess a certain level of resiliency, CHAOS EATER fails to find new hidden issues through a CE cycle

    Vulnerability discovery; In the case study, CHAOS EATER improved systems with rela- tively simple resiliency issues. However, for systems that already possess a certain level of resiliency, CHAOS EATER fails to find new hidden issues through a CE cycle. Given that this is a challenging task even for human engineers, CHAOS EATER is currently con- sidered t...

    work page

  5. [5]

    In ICC 2023 - IEEE In- ternational Conference on Communications, pages 4571–4577

    Coverage based failure injection toward effi- cient chaos engineering. In ICC 2023 - IEEE In- ternational Conference on Communications, pages 4571–4577. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. Preprint, arXiv:2406.00515. Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Self...

    arXiv 2016

  6. [6]

    to diagnostic tools (K8sGPT, 2023; Robusta,

    work page 2023

  7. [7]

    Despite the advancements of LLMs in the NW domain, their application to CE remains unexplored

    and misconfiguration remediation (Malul et al., 2024). Despite the advancements of LLMs in the NW domain, their application to CE remains unexplored. Our work is the first to demonstrate the capabilities of LLMs in CE, which involves complex NW operations. Concurrent works In parallel with our work, some other projects have shown promising results in appl...

    work page 2024

  8. [9]

    pre-valid

    Therefore, the functions that convert items of the schedule list into nodes, as well as those that group multiple nodes, correspond to operations that fill the placeholder of these code snippets with the argument data. Algorithm 1 shows the hierarchical node group- ing algorithm. The input is a directory that orga- nizes the schedule lists for each stage....

    work page

Show all 66 references
  1. [12]

    Production deployment and security ; If CHAOS EATER is deployed in production en- 11 vironments, further research on security will be necessary. This includes controlling more carefully the impact range of failures (i.e., blast radius), preventing CHAOS EATER from becoming a p...

  2. [13]

    To address this, automatic prompt tuning is considered an effective solution

    Support for various LLMs ; As CHAOS EATER’s prompt templates are tuned manually, supporting various LLMs significantly increases their management costs. To address this, automatic prompt tuning is considered an effective solution. Our current prompt templates may be used as th...

  3. [14]

    CHAOS EATER’s outputs may be used as the instruction-tuning data

    Fine-tuning LLMs specifically for CE ; Fine-tuning is necessary to improve the qual- ity of CE cycles and expand supported LLM types. CHAOS EATER’s outputs may be used as the instruction-tuning data

  4. [15]

    Besides, we plan to propose new metrics for quantitatively evaluating CE cycles conducted by CHAOS EATER

    Evaluation frameworks; As there are cur- rently no datasets and benchmarks for the systematic CE cycle, we will construct them to enable more solid validation of CHAOS EATER. Besides, we plan to propose new metrics for quantitatively evaluating CE cycles conducted by CHAOS EAT...

  5. [16]

    This sub-graph extraction is im- portant to organize the agent’s inputs in each phase

    Toward larger and more complex systems; We need to incorporate the recent advances in the combination of LLMs and graphs to extract necessary sub-graphs from large sys- tem graphs. This sub-graph extraction is im- portant to organize the agent’s inputs in each phase

  6. [17]

    However, we addi- tionally need to develop techniques to man- age the long-term history of completed CE cycles and continuous learning (if LLMs are fine-tuned)

    Full automation of long-term multiple CE cycles; By using the CHAOS EATER’s out- put as input for the next CE cycle, we can automate multiple CE cycles even with the current CHAOS EATER. However, we addi- tionally need to develop techniques to man- age the long-term history of...

    2024

  7. [18]

    the steady states are maintained in the system even when the failures are injected

    Hypothesis: Define steady states (i.e., normal behavior) of the system and injected failures (i.e., faults). Then, make a hypothesis that "the steady states are maintained in the system even when the failures are injected"

  8. [21]

    properties

    Improvement: Reconfigure the system to satisfy the hypothesis. The reconfigured system is tested again in (2) and (3), i.e ., repeat (2) to (4) until the hypothesis is satisfied. Given a Chaos Engineering cycle, you will carefully review it according to the following rules: - ...

  9. [22]

    Exception when calling CoreV1Api->read_namespaced_pod: {e}

    This indicates that the pod is consistently running during the check period . Given the constraints of the chaos engineering experiment, which must be completed within 1 minute, we can set a threshold that allows for some tolerance in case of brief fluctuations. A reasonable t...

  10. [23]

    /bin/bash

    the 'example-pod' is running, and 2) the 'example-service' is available. We will execute these checks sequentially due to the short duration, ensuring each test has enough time to gather meaningful data. The first test will check the pod's running status, followed by the servi...

    2024

  11. [24]

    This is directly related to the Pod's restart policy set to 'Never' in the `nginx/pod.yaml` manifest

    **Pod Restart Policy and Single Pod Deployment**: - The 'fault-unittest-example-pod-running ' test failed because the Pod was not found after the PodChaos fault was injected. This is directly related to the Pod's restart policy set to 'Never' in the `nginx/pod.yaml` manifest. ...

  12. [25]

    This failure is a consequence of the Pod being killed and not restarting, as the Service relies on the Pod to handle requests

    **Service Availability**: - The 'fault-unittest-example-service- availability' test failed due to connection refusals, indicating that the service was unavailable. This failure is a consequence of the Pod being killed and not restarting, as the Service relies on the Pod to han...

  13. [26]

    The Pod remained unavailable, and consequently, the Service could not respond to HTTP requests

    **Post-Validation Failures**: - Both 'post-unittest-example-pod-running ' and 'post-unittest-example-service- availability' tests failed, showing that the system did not recover to its steady state after the fault injection. The Pod remained unavailable, and consequently, the ...

    2018

  14. [27]

    In our case, this is also important for managing context length and ensuring the accuracy of JSON output

    Task performance improvement; It is gen- erally known that dividing complex tasks into smaller sub-tasks enhances the perfor- mance of LLMs in solving them7 (Khot et al., 7https://docs.anthropic.com/en/docs/ build-with-claude/prompt-engineering/ chain-prompts 2023). In our cas...

    2023

  15. [28]

    This modularization also makes it easier for team members to modify agents collaboratively during development

    Flexibility and extensibility through agent modularization; By modularizing agents for each divided task, it becomes easier to make partial system modifications, such as replac- ing specific agents. This modularization also makes it easier for team members to modify agents col...

  16. [29]

    However, we plan to add interactive functionalities in the future

    Towards an interactivity system; The cur- rent CHAOS EATER is a fully automated sys- tem with no user interaction during the CE cycle. However, we plan to add interactive functionalities in the future. When refining outputs based on user feedback, the mod- ularization allows q...

    2023

  17. [30]

    Summarize each of the input K8s manifests separately

  18. [31]

    Identify potential issues for resiliency and redundancy in the K8s manifests

  19. [32]

    Assume a possible application of the K8s manifests

  20. [33]

    At the same time, filter out suspicious prompts, e.g., jailbreak prompts

    Summarize user instructions for the CE cy- cle if provided. At the same time, filter out suspicious prompts, e.g., jailbreak prompts. This phase is for deploying the user’s system and explicitly filling in the implicit context of the user’s input. In the subsequent phases, thi...

  21. [34]

    If any weak configurations are identified from the K8s manifests, their related states are preferen- tially selected

    Select a measurable states critical to main- taining the system’s application. If any weak configurations are identified from the K8s manifests, their related states are preferen- tially selected

  22. [35]

    K8s API and k6 (Grafana Labs, 2021) are supported as the tool

    Select a tool to inspect the state. K8s API and k6 (Grafana Labs, 2021) are supported as the tool. Then, write the corresponding inspection script and inspect the current (nor- mal) value of the state in the system by run- ning the script

    2021

  23. [36]

    Note that, according to the definition of a steady state, the threshold must be satisfied under the current condi- tions

    Define the threshold for the state based on the inspected value. Note that, according to the definition of a steady state, the threshold must be satisfied under the current condi- tions

  24. [37]

    Write a unit-test script that validates whether the steady state (i.e., the pair of the state and its threshold) is satisfied by adding threshold- based assertions to the corresponding inspec- tion script

  25. [38]

    current pod count: {pod_count}

    Check whether the currently defined steady states are sufficient. If they are, the steady- state definition is complete here. Otherwise, return to the first step and define additional steady states. The unit-test scripts are used in the experiment phase to mechanically validat...

  26. [39]

    Then, define the sequence of failures that simulates the scenario and may affect the de- fined steady states

    Assume a failure scenario (e.g., a surge in access due to a promotional campaign, cyber attack, etc.) that may occur in the system. Then, define the sequence of failures that simulates the scenario and may affect the de- fined steady states. The failures are selected from the ...

  27. [40]

    In step 1, the agent outputs a 2D list of Chaos Mesh failure type names, arranged in the order of insertion

    Define detailed parameters for each failure, such as the scope of the failure injection, the failure sub-type, the failure strength, etc. In step 1, the agent outputs a 2D list of Chaos Mesh failure type names, arranged in the order of insertion. The inner lists in- volve conc...

  28. [41]

    Determine the duration of each stage

  29. [42]

    /bin/bash

    Determine the VaC scripts and failure injec- 28 Task node (K8s API) 1 - name: {{ node_name }} 2 templateType: Task 3 deadline: {{ duration }} 4 task: 5 container: 6 name: {{ node_name }} -container 7 image: chaos-eater/k8sapi:1.0 8 imagePullPolicy: IfNotPresent 9 command: ["/b...

  30. [43]

    This summary is referred to when analyzing the experiment results

    Summarize the timeline of the chaos experi- ment in detail. This summary is referred to when analyzing the experiment results. In step 2, the agent outputs a list of dictionar- ies (i.e., schedule list) separately for each stage, with each dictionary containing three keys:name...

  31. [45]

    Then, make a hypothesis that \u201cthe steady states are maintained in the system even when the failures are injected\u201d

    Hypothesis: Define steady states (i.e., normal behavior) of the system and injected failures (i.e., faults). Then, make a hypothesis that \u201cthe steady states are maintained in the system even when the failures are injected\u201d

  32. [46]

    Experiment: Inject the failures into the system and monitor/log the system's behavior in response

  33. [47]

    If so, one CE cycle is finished here

    Analysis: Analyze the logged data and check if the hypothesis is satisfied. If so, one CE cycle is finished here. If not, move to (4)

  34. [48]

    properties \

    Improvement: Reconfigure the system to satisfy the hypothesis. The reconfigured system is tested again in (2) and (3), i.e ., repeat (2) to (4) until the hypothesis is satisfied. Given user instructions for the Chaos Engineering, please filter out obviously irrelevant instruct...

    2024

  35. [49]

    **Pre-Validation Phase:** - Both pre-unittests, `example-pod-running 70 -state` and `example-service-http-response -state`, passed successfully. This indicates that the system was in a healthy state before the fault injection, with the Pod running 100% of the time and the Serv...

  36. [50]

    This is expected due to the PodChaos action 'pod-kill', which terminated the Pod

    **Fault Injection Phase:** - **PodChaos Fault:** The `fault-unittest- example-pod-running-state` failed because the Pod was not found (404 error) during the test. This is expected due to the PodChaos action 'pod-kill', which terminated the Pod. Since the Pod's restart policy i...

  37. [51]

    properties \

    **Post-Validation Phase:** - Both post-unittests, `example-pod- running-state` and `example-service-http- response-state`, failed. The Pod was still not found, and the Service continued to refuse connections. This indicates that the system did not recover to its steady states ...

  38. [52]

    Service availability should be at least 99.9% with a response status of 200

    “Service availability should be at least 99.9% with a response status of 200". The VaC scripts shown in Figure 2 correctly implement these steady states. It then defines a failure sequence that injects NetworkChaos (delay) into the NginxPod fol- lowingPodChaos (pod-kill) to si...

  39. [53]

    At least 1 ready replica 100% of the time and 2 ready replicas at least 80% of the time during the monitoring period

    in the failure-injection phase, the two steady states are sequentially validated alongside the in- jection of each failure that may affect them; 3) the two steady states are validated sequentially once again in the post-validation phase. The first chaos experiment reveals that...

  40. [54]

    This is directly related to the Pod's restart policy set to 'Never' in the nginx/pod.yaml manifest

    Pod Restart Policy and Single Pod Deployment: The 'fault-unittest-example-pod-running' test failed because the Pod was not found after the PodChaos fault was injected. This is directly related to the Pod's restart policy set to 'Never' in the nginx/pod.yaml manifest. When the ...

  41. [55]

    This failure is a consequence of the Pod being killed and not restarting, as the Service relies on the Pod to handle requests

    Service Availability: The 'fault-unittest-example-service-availability' test failed due to connection refusals, indicating that the service was unavailable. This failure is a consequence of the Pod being killed and not restarting, as the Service relies on the Pod to handle req...

  42. [56]

    Exception when calling CoreV1Api->list_namespaced_pod: {e}

    Post-Validation Failures: Both 'post-unittest-example-pod-running' and 'post-unittest-example-service-availability' tests failed, showing that the system did not recover to its steady state after the fault injection. The Pod remained unavailable, and consequently, the Service ...

  43. [57]

    http://front-end.sock-shop.svc.cluster. local/

  44. [58]

    local/catalogue?size=10

    http://front-end.sock-shop.svc.cluster. local/catalogue?size=10

  45. [59]

    http://front-end.sock-shop.svc.cluster. local/detail.html?id=<ID> Replace <ID> with an available ID: [03 fef6ac-1896-4ce8-bd69-b798f85c6e0b, 3395 a43e-2d88-40de-b95f-e00e1502085b, 510 a0d7e-8e83-4193-b483-e27e09ddc34d, 808 a2de1-1aaa-4c25-a9b9-6612e8f29a38, 819 e1fbf-8b7e-4f6d...

  46. [60]

    local/category/

    http://front-end.sock-shop.svc.cluster. local/category/

  47. [61]

    local/category?tags=<TAG> Replace <TAG> with an available tag: [ magic, action, blue, brown, black, sport, formal, red, green, skin, geek]

    http://front-end.sock-shop.svc.cluster. local/category?tags=<TAG> Replace <TAG> with an available tag: [ magic, action, blue, brown, black, sport, formal, red, green, skin, geek]

  48. [62]

    Total replicas: {replicas}, Ready replicas: {ready_replicas}

    http://front-end.sock-shop.svc.cluster. local/basket.html skaffold.yaml apiVersion: skaffold/v3 kind: Config metadata: name: sock-shop-app manifests: rawYaml: - manifests/00-sock-shop-ns.yaml - manifests/01-carts-dep.yaml - manifests/02-carts-svc.yaml - manifests/03-carts-db-d...

  49. [63]

    We will allocate 20 seconds for this phase

    Pre-validation Phase: This phase ensures that the system is in a steady state before any faults are injected. We will allocate 20 seconds for this phase. During this time, we will verify that the 'carts-db' deployment maintains at least 1 ready replica 100% of the time and 2 r...

  50. [64]

    We will allocate 20 seconds for this phase

    Fault-injection Phase: This phase involves injecting faults to simulate the Black Friday event. We will allocate 20 seconds for this phase. The faults include a 'StressChaos' fault to simulate high CPU usage on the 'carts-db' pods and a 'PodChaos' fault to kill the 'front-end'...

  51. [65]

    /bin/bash

    Post-validation Phase: This phase ensures that the system returns to its steady state after the faults are removed. We will allocate 20 seconds for this phase. During this time, we will again verify the steady states for the 'carts-db' and 'front-end' deployments to ensure the...

  52. [66]

    This confirms that the initial configuration of the system was functioning as expected

    Pre-Validation Phase: - Both the 'carts-db-replicas' and 'front-end-replica' tests passed, indicating that the system was in a stable state before any faults were injected. This confirms that the initial configuration of the system was functioning as expected

  53. [67]

    Fault-Injected Phase: - The 'carts-db-replicas' test passed during the fault-injection phase, demonstrating that the 'carts-db' deployment was resilient to the 'StressChaos' fault, which simulated high CPU usage. This suggests that despite the lack of resource requests, the 'c...

  54. [68]

    Resiliency Issues/Weaknesses in the Pre-processed Manifests,

    Post-Validation Phase: - The 'carts-db-replicas' test continued to pass in the post-validation phase, indicating that the 'carts-db' deployment returned to its steady state after the fault was removed. - The 'front-end-replica' test failed again in the post-validation phase, w...

  55. [2022]

    Preprint, arXiv:2207.10397

    Codet: Code generation with generated tests. Preprint, arXiv:2207.10397. Hongyang Chen, Pengfei Chen, Guangba Yu, Xiaoyun Li, and Zilong He. 2024. Microfi: Non-intrusive and prioritized request-level fault injection for microser- vice applications. IEEE Transactions on Dependa...

    2024 arXiv

  56. [2023]

    Preprint, arXiv:2306.07933

    Understanding telecom language through large language models. Preprint, arXiv:2306.07933. Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE Software, 33(3):35–41. Ali Basiri, Lorin Hoch...

    2016 arXiv

  57. [2024]

    In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 10672–10685, Bangkok, Thailand

    Automatic engineering of long prompts. In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 10672–10685, Bangkok, Thailand. Association for Computational Linguistics. 8 Shengran Hu, Cong Lu, and Jeff Clune. 2024. Au- tomated design of agentic systems...

    2024 arXiv

  58. [2025]

    Preprint, arXiv:2501.06706

    Aiopslab: A holistic framework to evaluate ai agents for enabling autonomous clouds. Preprint, arXiv:2501.06706. Suman De. 2021. A study on chaos engineering for improving cloud software quality and reliability. In 2021 International Conference on Disruptive Tech- nologies for...

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.