{"id":"de48a9bd-5adc-4801-be3c-04b9feae26f5","arxiv_id":"2505.13654","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study of 971 GitHub repositories shows Toxiproxy and Chaos Mesh are the most adopted chaos engineering tools, used mainly for development and network-fault testing.","lead":"Researchers analyzed 971 GitHub repositories that use 10 popular chaos engineering tools and mapped how, why, and where these tools are adopted. The study finds development teams dominate usage, network and instance-termination faults are the most tested, and Toxiproxy and Chaos Mesh lead adoption.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistent growth since 2016' claim plots repository creation dates, not tool-adoption dates; Toxiproxy repositories dated 2011 predate the tool's first commit in 2014, so the abstract's adoption-growth claim is not supported by Figure 2 without re-dating.","rationale":"The reader's weakest assumption—the absence of a GitHub-wide baseline for the growth analysis—is real and is explicitly conceded in Section 6 (Conclusion Validity). My stress-test identifies a prior, more direct threat to the same abstract claim: the temporal variable plotted in Figure 2 is the repository creation date, not the CE adoption date. The manuscript itself says adoption often happens long after repository creation, and the data contain Toxiproxy-using repositories created in 2011, before Toxiproxy's first commit in 2014. Because the growth curves are built from creation dates, they cannot by themselves support 'reflecting increasing adoption.' This is an internal-validity issue, not a disagreement with any field consensus, and it does not undermine the descriptive findings about which tools are used and for what purposes. The released dataset, the manual false-positive filtering, and the transparent reporting are genuine strengths. A conditional outcome remains appropriate: the authors can settle the concern by re-dating the growth analysis from commit history and normalizing with a GitHub baseline, or by softening the abstract's causal language to describe repository creation trends rather than adoption growth. The reader's verdict is therefore unchanged in direction, though the rationale is broadened.","tokens_in":20113,"tokens_out":5213,"duration_ms":48706,"concrete_test":"Re-analyze Figure 2 using, for each of the 971 repositories, the date of the first commit that introduces the CE tool (e.g., the first commit touching a file matched by that tool's search query in Table 1), and additionally normalize the resulting annual counts by the total number of GitHub repositories created in the same calendar year using GHTorrent or GitHub Archive. If the normalized, re-dated curves still show consistent growth from 2016 for Toxiproxy and from about 2020 for Chaos Mesh, the abstract claim stands. If not, the growth finding should be reworded as 'repositories created over time that eventually use these tools,' which is a different and weaker claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central adoption claim ('Toxiproxy and Chaos Mesh ... showing consistent growth since 2016 and reflecting increasing adoption') rests on Figure 2, whose caption says 'Yearly repository creation trends for the top 10 chaos engineering tools.' The x-axis is therefore the creation date of the GitHub repository, not the date the CE tool was first integrated. The paper itself notes in Section 4 (RQ1) that 'many repositories adopted these tools long after their initial creation.' The data make the problem concrete: Figure 2a shows Toxiproxy repositories created in 2011, yet Table 2 records Toxiproxy's first commit as 2014-09-07 and its first release later; those repositories cannot have adopted Toxiproxy at creation. The same logic applies to Chaos Mesh entries before its first commit date of 2019-09-04. Thus the growth curves mix two different temporal variables: when a repository was created and when CE was adopted. A repository created in 2011 that added Toxiproxy in 2024 is counted at 2011, making recent adoption invisible and early adoption overstated. Even if raw counts were normalized by GitHub-wide repository growth—as the authors themselves recommend in Section 6 (Conclusion Validity)—the normalized curve would still not measure adoption timing. Consequently, the 'consistent growth since 2016' finding is not established by the presented data, and the abstract's causal phrasing 'reflecting increasing adoption' goes beyond what the analysis supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a mining study of 971 GitHub repositories that incorporate one of ten popular chaos engineering (CE) tools. The authors select the ten tools from a compiled list of 41 using GitHub star counts and the number of importing repositories, query GitHub code search, remove false positives by manual inspection, and classify the remaining repositories by purpose (development, teaching, learning, research, unclassified) and by ownership origin. They then report descriptive statistics for five research questions: adoption trends over time (RQ1), tool release and commit activity (RQ2), purpose and ownership (RQ3), activity and popularity metrics (RQ4), and fault types exercised (RQ5). Headline findings are that Toxiproxy and Chaos Mesh are the most frequently used tools with consistent growth since 2016; that new tool releases peaked in 2018; that 58.0% of repositories serve development purposes; and that network (40.9%) and instance-termination (32.7%) faults dominate the injected scenarios.","tokens_in":20331,"tokens_out":7299,"duration_ms":60606,"significance":"The study is a useful descriptive contribution to the empirical software-engineering literature on tool adoption, following the established template of Sánchez et al. (2022). Its genuine strengths include a publicly available replication package on Zenodo, a manual verification step over roughly a thousand repositories, and an explicit validity section that acknowledges the main threats. The classification results (purpose, ownership, fault-type distribution) are grounded in the published dataset and will interest both practitioners and researchers. However, the headline adoption-growth claim is not supported by the data as analyzed: Figure 2 plots repository creation dates while the text and abstract interpret the curves as adoption, and the growth is not normalized by overall GitHub growth. These are load-bearing problems for the paper's central claim, though they appear fixable within scope by re-dating adoption events and reframing the claims.","major_comments":[{"comment":"The claim that Toxiproxy and Chaos Mesh show 'consistent growth since 2016' and 'increasing adoption' is not supported by the presented data. Figure 2's caption states that the plot shows 'Yearly repository creation trends,' so the x-axis records when each GitHub repository was created, not when the CE tool was adopted. The paper itself notes in §4 (RQ1) that 'many repositories adopted these tools long after their initial creation,' and Table 2 dates Toxiproxy's first commit to 2014-09-07 and Chaos Mesh's to 2019-09-04, yet Figure 2a includes Toxiproxy repositories created in 2011 and Chaos Mesh entries before 2019. A repository created in 2011 that added Toxiproxy in 2024 is counted at 2011, making recent adoption invisible and early adoption overstated. The abstract's phrase 'reflecting increasing adoption' therefore goes beyond what the analysis can establish; the authors should either re-date the analysis (e.g., to the first commit referencing the tool) or explicitly reword the RQ1 conclusion and abstract to refer to repository creation rather than adoption.","section":"§4 (RQ1), Fig. 2, Table 2; Abstract"},{"comment":"The growth curves are not normalized against the growth of GitHub as a whole. The authors acknowledge this in Section 6, stating that 'an observed increase in the number of projects using CE tools might reflect the general increase in GitHub activity over time, rather than a genuine increase in interest or adoption of CE practices.' This caveat is correctly placed, but it is not a mere validity footnote: the unnormalized trend is the basis of the abstract's headline 'consistent growth since 2016' claim. Moreover, normalizing by repository-creation counts would still not fix the deeper problem that Figure 2 measures creation dates rather than adoption dates (see previous comment). Both corrections are needed before the adoption-growth finding can stand as stated.","section":"§4 (RQ1), §6 (Conclusion Validity)"},{"comment":"The finding that 'Toxiproxy and Chaos Mesh are the most frequently used' (§4, Abstract) is partly determined by the tool-selection criterion. Section 3 states that the ten tools were chosen using 'the number of GitHub repositories that import the tools' and their star counts, and the same import-count measure then underlies the prevalence ranking reported in Table 1 and RQ1. The ranking is not fully tautological because the final counts come after false-positive filtering, which could reorder the tools, but the paper currently presents the prevalence result as an independent discovery ('The analysis reveals that...') without discussing its dependence on the selection procedure. The authors should acknowledge this circularity explicitly and frame the prevalence ranking as a property of the chosen sampling frame, or demonstrate that the post-filtering ranking is robust to reasonable variations of the selection rule.","section":"§3 (Tool Selection), §4 (RQ1), Abstract"}],"minor_comments":[{"comment":"The statement that the authors 'excluded 1,248 false positives, leaving 971 verified repositories in the dataset' is numerically inconsistent with Table 1, which reports 5,845 candidate repositories and 971 retained, implying 4,874 exclusions; the source of the 1,248 figure should be clarified or corrected.","section":"§6 (Internal Validity)"},{"comment":"The last column header, '# Repos w/ FP)', appears malformed and its meaning is unclear; the caption should state explicitly that this column reports the final number of repositories retained after false-positive removal.","section":"Table 1"},{"comment":"In the manuscript as submitted, the labels of Figures 4 and 6 render as unreadable 'uni000000...' character sequences rather than readable category names (e.g., 'Teaching,' 'Industry'); the encoding or figure generation must be fixed before the paper can be evaluated visually.","section":"Figures 4 and 6"},{"comment":"The phrase 'consistent growth since 2016' conflicts with Figure 2a, which shows Toxiproxy repositories from 2011 onward and Chaos Mesh only from 2019; the wording should be aligned with whatever variable the corrected figure actually plots.","section":"Abstract and §4 (RQ1)"},{"comment":"The tool name is written inconsistently as both 'Kube-monkey' (text, Table 1) and 'Kube-Monkey' (Figure 2b, Figure 8); one canonical spelling should be used throughout.","section":"Throughout"},{"comment":"The RQ5 summary box contains the typo 'Chaos enginnering'; this should be corrected.","section":"§4 (RQ5)"},{"comment":"The two-reviewer classification procedure is described, but no inter-rater agreement statistic (e.g., Cohen's kappa) is reported for the 971-repository classification, which would strengthen the internal-validity argument.","section":"§3 (Manual Identification)"}],"recommendation":"major_revision","confidential_remarks":"The abstract's adoption-growth sentence is the part of the paper most likely to be cited, and it rests on the creation-date curves in Figure 2; the authors already acknowledge the normalization problem in Section 6 but did not let that caveat temper the abstract's wording. I recommend requiring the re-dating/normalization fix, or a substantial rewording of the abstract and RQ1, before the paper is accepted, rather than accepting it as a purely descriptive study. The numeric inconsistency about the number of false positives and the garbled figure labels suggest the manuscript needs a careful production pass as well. The underlying dataset and replication package are real assets and make the required re-analysis feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Readable, transparent repository-mining study, and the descriptive core is solid. The authors built a real dataset from 971 GitHub repositories that use ten chaos engineering tools, manually reviewed false positives, and published the data and scripts on Zenodo. The counts of which tools are used, for what purpose (58% development), by whom (43% industry), and which fault types are exercised (network 41%, instance termination 33%) are useful and reasonably well supported. This looks like the first large-scale map of CE tool adoption on GitHub, and it deserves credit for that.\n\nThe soft spots are real but concentrated. The abstract's headline claim that Toxiproxy and Chaos Mesh show \"consistent growth since 2016 and reflecting increasing adoption\" is not established by the data as presented. The stress-test note is correct: Figure 2's x-axis is repository creation date, not adoption date. Toxiproxy repositories dated 2011 cannot have used the tool then, since its first commit was 2014; those are repositories created earlier that added the tool later. So the growth curves mix creation timing with adoption timing, and the abstract's causal phrasing goes beyond the evidence. The authors do flag the unnormalized-GitHub-growth issue in Section 6, but they never fix the deeper timing problem. At minimum the abstract and RQ1 conclusion need rewording, or better, re-analysis using the date the tool first appears in the repo.\n\nThe tool selection also introduces a mild circularity: the ten tools were chosen by GitHub stars and import counts, so finding that those tools dominate is partly a consequence of the selection. That doesn't sink the descriptive findings, but it means \"most used\" is not an independent discovery.\n\nMinor items: the threats section mentions RQ4 where it clearly means RQ5; inter-rater agreement is described qualitatively rather than quantified; and the false-positive rate is high (971 of 5,845), which makes the manual review procedure important enough to document more precisely. These are fixable.\n\nNet: solid empirical contribution with a load-bearing weakness in one advertised finding. I'd send it to peer review with a required revision on the growth analysis and abstract. The descriptive dataset is worth citing.","headline":"Useful descriptive dataset and honest reporting, but the abstract's 'consistent growth since 2016' adoptions claim is not actually supported because Figure 2 plots repository creation dates, not when the tool was adopted.","tokens_in":20925,"tokens_out":1591,"would_cite":true,"duration_ms":17449,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes an empirical picture of chaos engineering tool adoption by mining 971 GitHub repositories, showing that use is concentrated in Toxiproxy and Chaos Mesh, that network and instance-termination faults dominate…","keywords":["chaos engineering","fault injection","GitHub repository mining","tool adoption","resilience testing","cloud-native systems","Toxiproxy","Chaos Mesh"],"falsifier":"Re-run the authors' tool queries by creation year and divide by the total number of GitHub repositories created in that year; if the normalized share of CE-using repositories is flat or declining after 2016, the paper's 'consistent growth' conclusion is an artifact of platform growth.","tokens_in":19872,"feed_emoji":"💥","tokens_out":7322,"duration_ms":64900,"temperature":0.7,"pith_summary":"The paper aims to show, from public evidence, that chaos engineering has become a recognizable, tool-mediated practice in open source: a small set of tools accounts for most usage, new tool creation peaked around 2018 and then gave way to integration and refinement, and practitioners overwhelmingly inject infrastructure-level faults rather than application-level ones. A sympathetic reader should care because these findings give concrete grounding for deciding which resilience-testing tools, research directions, and fault categories deserve attention. The study also exposes a gap: research and teaching repositories are a minority, while industry-driven development repositories dominate the most active projects.","feed_headline":"Toxiproxy and Chaos Mesh dominate GitHub chaos tooling","feed_subtitle":"A 971-repository study finds chaos testing is mostly network and pod failures, with app-level faults at just 3 percent.","key_machinery":"The load-bearing mechanism is a mining-and-filtering pipeline: select ten chaos engineering tools by literature presence and GitHub popularity; query GitHub code search for tool-specific integration signatures in configuration, code, and build files; retrieve repository metadata through the GitHub API; manually remove false positives, reducing 5,845 candidate repositories to 971; then classify each repository by purpose and owner origin and count concrete fault-type usages against a four-category taxonomy. This pipeline converts general claims about adoption into countable, auditable evidence.","core_discovery":"The central discovery is an empirical map of chaos engineering tool usage drawn from 971 manually verified GitHub repositories across ten popular tools. The study claims that Toxiproxy and Chaos Mesh are the most widely used tools, with steady growth since 2016; that the ecosystem's release activity peaked in 2018 and then shifted toward refinement and integration, led by Chaos Mesh and LitmusChaos; that development-oriented repositories dominate at 58.0%, with industry ownership at 43.2%; and that tested fault scenarios are concentrated in network disruptions (40.9%) and instance termination (32.7%), with application-level faults at only 3.0%.","pith_inferences":["The 3.0% application-fault figure likely understates real usage: fault injections in application code are often written as custom scripts or test frameworks, which the tool-name-based search would miss; a signature-based search for exception injection and method-call-delay patterns could test whether the gap is real or an artifact.","Because teaching and learning repositories are numerous but low-commit, the pipeline probably captures a large population of tutorial users; an implication the authors do not draw is that educational content is a primary adoption channel, so improving tutorials may matter more than adding features.","The observed growth since 2016 could be tested directly by re-running the same queries per year and dividing by the total number of GitHub repositories created that year; if the normalized fraction is flat, the headline should be rewritten as 'chaos tooling grew with GitHub,' not 'chaos tooling grew.'","The findings suggest a research opportunity: since industry-owned repositories dominate while academic research repositories are only 5.7%, direct interviews or surveys with maintainers of Toxiproxy- and Chaos Mesh-using repositories could reveal why tools are chosen and why application-level faults are skipped; repository mining alone cannot answer that."],"forward_implications":["If the counts are representative, Toxiproxy and Chaos Mesh are de facto standards for network-level and Kubernetes-native chaos, and new tools should target integration or specific gaps rather than general fault injection.","If release activity peaked in 2018, the field's center of gravity has moved from creating tools to maintaining and embedding them; release frequency and commit recency, not star counts alone, now separate active from dormant tools.","If 81% of high-commit repositories are development-focused, industrial use is the main driver of deep, sustained adoption, while teaching and learning repositories are typically short-lived experiments rather than production usage.","If application-level faults are only 3.0% of observed fault scenarios, there is a concrete, measurable niche for tools and research on exceptions, return-value modification, and method-call delay.","Adoption-trend conclusions should be read as provisional until normalized by overall GitHub repository growth, because the paper's own validity section states that raw increases may merely track platform growth."],"supporting_citations":[{"why":"Supplies the repository-mining methodology and the purpose/activity classification scheme the study adapts.","marker":"(Sánchez et al, 2022)"},{"why":"Supplies the parallel data-quality-tool mining workflow and the classification review process.","marker":"(Ustunboyacioglu et al, 2024)"},{"why":"The authors' earlier multivocal review of chaos engineering; supplies the initial 40-tool list and the taxonomy used for tool selection.","marker":"(Owotogbe et al, 2024)"},{"why":"Supports using GitHub stars as a popularity criterion and describes factors that impact repository popularity.","marker":"(Borges et al, 2016)"},{"why":"Documents the promises and perils of mining GitHub, used in the external-validity argument about platform coverage.","marker":"(Kalliamvakou et al, 2016)"},{"why":"Documents Chaos Mesh's hosted-foundation status and maturity level, used to argue community engagement.","marker":"(Cloud Native Computing Foundation, 2025a)"},{"why":"Supports the conclusion-validity caveat that observed GitHub trends may reflect platform growth rather than genuine adoption.","marker":"(Cosentino et al, 2016)"}],"fun_headline_variants":["Chaos Mesh and Toxiproxy lead GitHub chaos tooling","Chaos testing on GitHub: network faults 41%, app faults 3%","971 GitHub repos: chaos engineering skews to dev, not apps","Chaos tooling study: Toxiproxy and Chaos Mesh rise since 2016","GitHub chaos repos reveal app-level fault testing gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that chaos engineering tool adoption has grown since 2016 rests on the assumption that rising counts of CE-using repositories reflect rising adoption, not simply the overall growth of GitHub; the paper does not normalize for the number of repositories created each year.","fun_headline_variants_meta":{"raw":{"variants":["Chaos Mesh and Toxiproxy lead GitHub chaos tooling","Chaos testing on GitHub: network faults 41%, app faults 3%","971 GitHub repos: chaos engineering skews to dev, not apps","Chaos tooling study: Toxiproxy and Chaos Mesh rise since 2016","GitHub chaos repos reveal app-level fault testing gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3240,"prompt_tokens":881,"completion_tokens":2359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":2265}},"tokens_in":497,"tokens_out":2359,"duration_ms":16632,"temperature":1.0,"reasoning_tokens":2265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:11:41.691506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the authors' tool queries by creation year and divide by the total number of GitHub repositories created in that year; if the normalized share of CE-using repositories is flat or declining after 2016, the paper's 'consistent growth' conclusion is an artifact of platform growth.","supporting_citations":[{"cited_title":"In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pp 120--129","cited_arxiv_id":null,"evidence_quote":"Supplies the parallel data-quality-tool mining workflow and the classification review process."},{"cited_title":"arXiv preprint arXiv:241201416","cited_arxiv_id":null,"evidence_quote":"The authors' earlier multivocal review of chaos engineering; supplies the initial 40-tool list and the taxonomy used for tool selection."},{"cited_title":"In: 2016 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 334--344","cited_arxiv_id":null,"evidence_quote":"Supports using GitHub stars as a popularity criterion and describes factors that impact repository popularity."},{"cited_title":"Empirical Software Engineering 21:2035--2071","cited_arxiv_id":null,"evidence_quote":"Documents the promises and perils of mining GitHub, used in the external-validity argument about platform coverage."},{"cited_title":"In: Proceedings of the 13th International Conference on Mining Software Repositories, pp 137--141","cited_arxiv_id":null,"evidence_quote":"Supports the conclusion-validity caveat that observed GitHub trends may reflect platform growth rather than genuine adoption."}],"review_version":1}