Pith. sign in

REVIEW 4 major objections 6 minor 82 references

Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that flaky tests routinely spread across project boundaries in the OpenStack ecosystem, affecting 55% of its Zuul-based projects and measurably slowing code review.

desk verdict A credible first look at ecosystem-level flakiness whose headline magnitudes need validation of the shared-name assumption before I'd cite the numbers. read the letter →

arxiv 2602.09311 v2 pith:FKTQZPXY submitted 2026-02-10 cs.SE

Tao Xiao , Dong Wang , Shane McIntosh , Hideaki Hata , Yasutaka Kamei This is my paper · ORCID
classification cs.SE
keywords flakytestscross-projectflakinessOpenStackcontinuousintegrationZuulCIcodereviewtestscopesinconsistent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that test flakiness is not just a per-project problem: in a large interconnected CI ecosystem, a single non-deterministic test can fail in many projects at once. Analyzing a year of OpenStack code-review and CI data, it reports 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests, with 55% of Zuul-using projects touched. It argues that cross-project spread is more frequent and more costly than single-project flakiness, and that even unit tests—supposedly isolated—appear in 70% of cross-project cases. A sympathetic reader would care because ecosystem-level flakiness shifts where mitigation effort belongs: shared CI jobs and standardized configurations, not just individual test code.

What carries the argument

The load-bearing object is the shared CI artifact identity: a Zuul job name (coarse-grained) or a fully-qualified test name (fine-grained) that recurs across multiple OpenStack projects. The paper treats that recurrence as evidence that the same job definition or test code is shared through inheritance (for example, the devstack-tempest parent job), then labels the recurrence cross-project flakiness. Flakiness levels are built from the matrix of which projects run a given test and which ones see it fail: Shared-Partial Trigger, Uniform Multi-Project, Cross-Trigger, and Project-Isolated. This classification carries the prevalence, cost, and cause arguments.

What would settle it

Pick the 1,535 cross-project flaky tests, fetch the source of each named test in each project, and compare code plus Zuul job definitions. If a substantial share (say, more than 10%) of same-named tests turn out to be different implementations or locally overridden jobs, the cross-project interpretation collapses to a naming artifact. A cheaper first check: inspect the inheritance graph of the 371 cross-project flaky jobs to confirm each job's definition is literally shared, not coincidentally named.

Watch

Extended reading notes

Core claim

The central discovery is that flaky tests in OpenStack frequently operate across project boundaries: same-named tests and shared CI jobs fail non-deterministically in multiple projects, not just one. Combining rerun detection in code-review comments with build-log analysis over June 2023–June 2024, the authors identify 29,911 flaky builds, 1,535 cross-project flaky tests, and 1,105 inconsistently flaky tests; 378 projects (about 55% of the 691 that use Zuul CI) are affected. Cross-project flaky builds waste more review time than project-isolated ones (significant at p<0.001), and 70% of flaky unit tests show uniform multi-project behavior, contradicting the assumption that unit tests are ins

Load-bearing premise

The counts of cross-project flakiness assume that two projects running the same Zuul job name or the same fully-qualified test name are running the same shared job definition and the same test code; if those names merely collide across forks or locally overridden definitions, the 55% figure and the 1,535/1,105 counts include artifacts of naming.

Editorial extensions

If this is right

  • If flaky tests propagate across projects, fixing them becomes a coordination problem: one shared job failure can block reviews in many projects at once.
  • Standardizing CI configurations and pinning dependencies across the ecosystem could remove a large share of inconsistent flakiness, since configuration and dependency mismatches are among the top identified causes.
  • The 'recheck and wait' paradigm amplifies waste; replacing it with early intervention on flaky tests and flaky jobs would save review time and CI resources.
  • Unit tests being cross-project flaky implies that test isolation, mocking, and dependency injection need rethinking even for unit tests reused across projects.
  • Other large CI ecosystems can use the 55% figure and the 1,535/1,105 counts as a baseline when measuring their own cross-project flakiness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same-name identity assumption holds only weakly, the true rate of shared flakiness may be lower; a replication that verifies job inheritance and test-code equality across projects would sharpen or shrink the estimate. This is an inference about the paper's load-bearing premise, not a paper claim.
  • The four-level flakiness taxonomy could be reused as a monitoring signal: a test shifting from Project-Isolated to Shared-Partial Trigger would indicate a shared resource or environment degrading. The paper does not propose this use.
  • Since 89% of inconsistent flakiness is event-related, many 'flaky tests' may really be flaky environments; if so, ephemeral containerized CI environments with pinned resources could remove a large share of cross-project flakiness without touching test code. The paper suggests environment standardization but does not test it.
  • The paper's time-waste result (1,156 days across all flaky builds) implies that ecosystem-level CI coordination has direct economic consequences; extending the same measurement to other ecosystems would reveal whether OpenStack is typical or extreme.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents an empirical study of flaky tests in the OpenStack ecosystem, focusing on two new phenomena: cross-project flakiness (the same flaky test or CI job affecting multiple projects) and inconsistent flakiness (a test flaky in some projects but stable in others). The authors collect one year of Gerrit/Zuul CI data (June 2023–June 2024), identify 29,911 flaky builds from repeated builds on the same patch set, and then drill down to test-level results for a subset of those builds. They report that cross-project flakiness affects 55% of OpenStack projects, identify 1,535 cross-project and 1,105 inconsistently flaky tests, find that 70% of unit tests exhibit cross-project flakiness, and qualitatively attribute inconsistent flakiness to event-related, dependency-related, and configuration-related causes. They also quantify extra review delay and gather developer feedback through a questionnaire. A replication package is provided.

Significance. If the headline claims hold, this is one of the first ecosystem-level studies of flakiness and would be a useful contribution to the flaky-test and CI literature. The paper has real strengths: a public replication package, test-level analysis in addition to build-level analysis, a manually coded qualitative dataset with reported inter-rater agreement (Fleiss kappa = 0.726), statistical tests with effect sizes, and a small external validation on the Zuul ecosystem. However, the quantitative claims currently outrun the evidence in two ways: the identity-by-name assumption for jobs and tests is asserted rather than verified, and the headline test counts are derived from a heavily filtered subset without making that clear in the abstract. The core existence claim—that flaky tests and jobs recur across project boundaries in OpenStack—is plausible and supported by the data, but the specific prevalence numbers need substantial validation and re-framing before they can be taken at face value.

major comments (4)
  1. [V-A, VI-A, X] The paper operationalizes cross-project flakiness by checking whether the same Zuul job name (Section V-A) or the same fully-qualified test name (Section VI-A) recurs across projects. Section VI-A asserts that this identifier 'ensures tests with this identifier correspond to semantically equivalent code,' but no verification is reported. In OpenStack, two projects can define jobs with the same name but different parents, parameters, or local bodies, and the same fully-qualified test name can appear in forks or vendored copies. Section X only discusses misclassification due to insufficient collection, not name collisions. This is load-bearing because Observation 1 (378 projects, 55%) and Observation 4 (1,535/1,105 tests) are the paper's headline results. Please either verify shared definitions via Zuul config parent chains and test source identity, quantify the collision rate, or soften t
  2. [VI-B, Table II, Abstract] Observation 4's counts of 1,535 cross-project and 1,105 inconsistent flaky tests are not counts of all flaky tests. They are sums from the bottom panel of Table II, which is based on the 2,403 tests that remain after excluding single-patch-set tests (Section VI-B). The top panel of Table II counts 11,506 flaky tests, so the two panels use different denominators. The abstract presents 1,535/1,105 as global totals, and Observation 5's '70% of unit tests' is likewise conditional on the filtered set. This is misleading. Please report both raw and filtered counts in the abstract and results, or explain why the filter is appropriate for the headline claim.
  3. [V-B, Observation 2] Observation 2 reports a growth from 52 affected projects in 2020 to 378 in 2024, but Section IV-B states that data collection covers June 2023–June 2024. It is not explained how 2020–2022 data were obtained. If the one-year dataset was used to construct the time series, the pre-2023 portion is unsupported; if data from a prior study were merged, that source and the different collection methods need to be described. This result underpins the conclusion that cross-project flakiness has 'rapidly accumulated over time' and requires either proper longitudinal data or removal.
  4. [VI-A] The test-level analysis is based on 3,369 of 29,911 flaky builds (11%), spanning 370 of 1,651 unique jobs. The paper does not analyze whether the retrievable subset differs systematically from the non-retrievable subset (e.g., by project, job, or retention policy). If log retention is not independent of flakiness, the 1,535/1,105 counts and Observation 5's scope percentages may be biased. Please compare retrievable versus non-retrievable builds/jobs/projects on available observables and discuss the selection bias risk.
minor comments (6)
  1. [IV-A] Typo: 'prorogation' should be 'propagation'.
  2. [VI-B] The sentence 'This filtering yields 2,403 flaky tests from 11,506 flaky instances' is confusing; it should likely read 'from 11,506 flaky tests' or 'from 57,124 flaky instances.'
  3. [Figure 2] The left and right axes are not labeled. The left axis shows frequency of flaky builds and the right axis shows number of projects, but this is not stated in the caption.
  4. [Table II] The top (test scope) and bottom (flakiness level) panels use different test populations (11,506 vs 2,403). Add a note explaining the difference and which observations rely on which panel.
  5. [VIII] Observation 8 refers to 'additional build execution time,' but the baseline for 'additional' is not explicitly defined. Please state precisely how the time waste for a flaky build is measured.
  6. [X] The external-validation claim is based on only six Zuul projects, one cross-project job, and three inconsistent flaky tests. The phrase 'confirm the applicability of our results in Zuul' is stronger than this evidence supports; please temper the wording.

Circularity Check

1 steps flagged · score 4.0 of 10

Peripheral RQ3 flaky-range confirmation is definitional, but core cross-project prevalence counts are independently measured.

  1. self definitional [Section VII-B, Observation 7, Figure 4]
    "We define the flaky range as the duration from the first identified instance of flakiness to the latest recorded instance in our data set for each flaky job. This range provides perspective on the persistence of flaky behavior over time and across projects. We then examine differences in failure rates within, before, and after the flaky range of each job by performing a Mann-Whitney test [36] (two-tailed, unpaired, α= 0.05)."

    The flaky range is not an independent time window; its endpoints are defined by the first and latest identified flaky instances, i.e., by the failing test executions that are the outcome being counted. Every identified flaky failure falls inside the interval by construction, while the before/after intervals are defined to contain no identified flaky instance. Comparing failure rates inside vs. outside an outcome-selected interval therefore re-expresses the selection rule rather than testing it; the reported significant difference is an artifact of the partition, not independent evidence that race-condition flakiness elevated failure rates.

full rationale

The core derivation chain is otherwise non-circular. Flaky builds are detected from CI comments as opposing outcomes on the same patch set (Section IV-B), test-level flakiness is retrieved from build logs (Section VI-A), and the 55%, 1,535, and 1,105 figures are descriptive summaries of an independently scraped one-year corpus, not fitted parameters renamed as predictions. The identity-by-name rule for jobs/tests (Sections V-A, VI-A) is a construct-validity threat (name collisions, locally overridden job definitions), not a circularity, because the phenomenon is conceptually defined as cross-project propagation and name recurrence is only an operational proxy; no outcome is fit from that proxy. The self-citation to [34] is contextual prior work with independent data, not a load-bearing uniqueness assumption. Section X transparently discloses the main validity limits (misclassification due to insufficient collection; no cross-validation of RQ3 causes, with risk of misattributed bug reports). These affect correctness and generalizability, not circularity. The flaky-range confirmation is localized to RQ3 and does not undermine the main prevalence/time-waste observations, hence score 4.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper's central quantities rest on several hand-chosen thresholds and unverified identity assumptions: the 200-build minimum per job, the single-patch-set exclusion, the 'unit'/'api'/... keyword taxonomy, and the saturation stopping rule for coding. None of these is fitted to data in the mathematical sense, but each shapes the headline numbers, so they are listed as free choices. The axioms are the domain assumptions that make the measurement meaningful; the most fragile is that same-named jobs/tests across projects denote the same shared software artifact.

free parameters (4)
  • Minimum builds per flaky job (200)
    Hand-chosen threshold in Section V-A; jobs with fewer than 200 builds are excluded from the cross-project analysis, shaping the 55% and job-level counts.
  • Qualitative saturation criterion (10)
    Coding stops after 10 consecutive inconsistently flaky tests with no new codes (Section VII-A); 110 of 1,105 tests are coded, so the cause frequencies (98/23/23) inherit this stopping rule.
  • Single-patch-set exclusion filter
    Flaky tests appearing in only one patch set are excluded from Table II and the lower-bound counts; this converts 11,506 tests into 2,403 and determines the 1,535/1,105 headline numbers (Section VI-B).
  • Test-scope keyword lists
    Fully-qualified class-name substrings ('unit','api','scenario','functional','fullstack') are iteratively hand-refined (Section VI-A); the 'unit' label drives the 70% unit-test claim.
assumptions (6)
  • domain assumption Zuul CI is representative of OpenStack CI; only Zuul-using projects are analyzed
    Section IV-B: 691/1,336 OpenStack projects use Zuul; the 55% denominator is 691 Zuul projects, yet conclusions are phrased about 'the OpenStack ecosystem'.
  • domain assumption A failure followed by a success on the same patch set identifies a flaky build
    Section IV-B 'Flaky Build Identification'; inherits the recheck-based definition from Maipradit et al. [34] (overlapping authorship) without independent validation on 2023–2024 data.
  • domain assumption Identical fully-qualified test identifiers across projects denote semantically equivalent code
    Section VI-A 'Flaky Test Identification'; asserted to prevent naming collisions, but code equivalence is not verified (forks or unrelated classes can share a dotted name).
  • domain assumption Same Zuul job name across projects indicates a shared job definition
    Section V-A; no check that recurring jobs inherit from a common parent (the Figure 1 example shows inheritance, but it is not verified for all 371 cross-project jobs).
  • standard math Mann-Whitney test assumptions (independence) hold for per-job failure rates before/within/after the flaky range
    Section VII-B; repeated measures from the same jobs are treated as independent samples, which the test does not guarantee.
  • domain assumption Theoretical saturation at 10 consecutive new-code-free tests yields a complete cause taxonomy
    Section VII-A coding saturation; analytic generalization on 10% of inconsistent tests assumes the 145 codes cover all important causes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem." pith.science (2026). https://pith.science/paper/FKTQZPXY

@misc{pith2026260209311,
  author       = {Pith},
  title        = {Pith review of: Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKTQZPXY}},
  note         = {Machine review of arXiv:2602.09311}
}
read the original abstract

Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies.

Figures

Figures reproduced from arXiv: 2602.09311 by the authors.

Figure 1
Figure 1. Real-world example of cross-project flakiness during the code review #877934 (Cinder project) and #882133 (Glance [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. shows that cross-project flakiness demonstrates a faster growth rate over time compared to those not. This trend highlights the increasing complexity of managing CI workflows as projects evolve to use shared resources more extensively. The frequent occurrence of cross-project flakiness also emphasizes the need for robust cross-project coordination to address these issues effectively, as unmanaged flakiness in shared… view at source ↗
Figure 3
Figure 3. Parallel sets between test scopes and flakiness levels [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Failure rates of each Zuul job before, within, and after [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Time wasted (hours) of each Zuul job. TABLE IV: Respondents’ role and their experiences in Open￾Stack as well as with cross-project and inconsistent flakiness. ID Role Experience OpenStack Cross-project Inconsistent R1 Contributor 3-5 years No Yes R2 Core developer Mor…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 1 linked inside Pith

  1. [1]

    An empirical study of integration activities in distributions of open source software,

    B. Adams, R. Kavanagh, A. E. Hassan, and D. M. German, “An empirical study of integration activities in distributions of open source software,”Empirical Soft- ware Engineering, vol. 21, pp. 960–1001, 2016

    2016

  2. [2]

    230,439 test failures later: An empirical evaluation of flaky failure classifiers,

    A. Alshammari, P. Ammann, M. Hilton, and J. Bell, “230,439 test failures later: An empirical evaluation of flaky failure classifiers,” in2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2024, pp. 257–268

    2024

  3. [3]

    Flakeflagger: Predicting flakiness without rerunning tests,

    A. Alshammari, C. Morris, M. Hilton, and J. Bell, “Flakeflagger: Predicting flakiness without rerunning tests,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1572– 1584

    2021

  4. [4]

    Deflaker: Automatically detecting flaky tests,

    J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “Deflaker: Automatically detecting flaky tests,” inProceedings of the 40th international confer- ence on software engineering, 2018, pp. 433–444

    2018

  5. [5]

    Study- ing the impact of adopting continuous integration on the delivery time of pull requests,

    J. H. Bernardo, D. A. da Costa, and U. Kulesza, “Study- ing the impact of adopting continuous integration on the delivery time of pull requests,” inProceedings of the 15th International Conference on Mining Software Repositories, 2018, pp. 131–141

    2018

  6. [6]

    How to break an api: cost negotiation and community values in three software ecosystems,

    C. Bogart, C. K ¨astner, J. Herbsleb, and F. Thung, “How to break an api: cost negotiation and community values in three software ecosystems,” inProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016, pp. 109– 120

    2016

  7. [7]

    Test verification,

    Brown, Geoff, “Test verification,” https://web.archive. org/web/20201022110253/https://developer.mozilla. org/en-us/docs/mozilla/qa/test verification, [Accessed 17-02-2025]

    arXiv 2025

  8. [8]

    Rechecking recheck requests in continuous integration,

    Y . Brus, R. Maipradit, E. T. Barr, and S. McIntosh, “Rechecking recheck requests in continuous integration,” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2025, p. To appear

    2025

Show all 82 references
  1. [9]

    The silent helper: the impact of continuous integration on code reviews,

    N. Cassee, B. Vasilescu, and A. Serebrenik, “The silent helper: the impact of continuous integration on code reviews,” in2020 IEEE 27th International Confer- ence on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2020, pp. 423–434

    2020

  2. [10]

    Charmaz,Constructing Grounded Theory

    K. Charmaz,Constructing Grounded Theory. SAGE, 2014

    2014

  3. [11]

    Dominance statistics: Ordinal analyses to an- swer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to an- swer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  4. [12]

    The social side of software platform ecosystems,

    C. R. de Souza, F. Figueira Filho, M. Miranda, R. P. Ferreira, C. Treude, and L. Singer, “The social side of software platform ecosystems,” inProceedings of the 2016 CHI conference on human factors in computing systems, 2016, pp. 3204–3214

    2016

  5. [13]

    An empirical compar- ison of dependency issues in oss packaging ecosystems,

    A. Decan, T. Mens, and M. Claes, “An empirical compar- ison of dependency issues in oss packaging ecosystems,” in2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER). IEEE, 2017, pp. 2–12

    2017

  6. [14]

    Empirical study of restarted and flaky builds on travis ci,

    T. Durieux, C. Le Goues, M. Hilton, and R. Abreu, “Empirical study of restarted and flaky builds on travis ci,” inProceedings of the 17th International Conference on Mining Software Repositories, 2020, pp. 254–264

    2020

  7. [15]

    Understanding flaky tests: The developer’s perspective,

    M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: The developer’s perspective,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering, 2019, pp. 830–840

    2019

  8. [16]

    Building theories from case study research,

    K. M. Eisenhardt, “Building theories from case study research,”Academy of management review, vol. 14, no. 4, pp. 532–550, 1989

    1989

  9. [17]

    Flakify: A black-box, language model-based predictor for flaky tests,

    S. Fatima, T. A. Ghaleb, and L. Briand, “Flakify: A black-box, language model-based predictor for flaky tests,”IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 1912–1927, 2022

    1912

  10. [18]

    Measuring nominal scale agreement among many raters

    J. L. Fleiss, “Measuring nominal scale agreement among many raters.”Psychological bulletin, vol. 76, no. 5, p. 378, 1971

    1971

  11. [19]

    Release synchronization in software ecosystems: Empirical study on openstack,

    A. Foundjem and B. Adams, “Release synchronization in software ecosystems: Empirical study on openstack,” Empirical Software Engineering, vol. 26, pp. 1–50, 2021

    2021

  12. [20]

    The importance of accounting for execution failures when predicting test flakiness,

    G. Haben, S. Habchi, J. Micco, M. Harman, M. Pa- padakis, M. Cordy, and Y . Le Traon, “The importance of accounting for execution failures when predicting test flakiness,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engi- neering, 2024, pp...

    2024

  13. [21]

    The art of testing less without sacrificing quality,

    K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy, “The art of testing less without sacrificing quality,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1. IEEE, 2015, pp. 483–493

    2015

  14. [22]

    The review linkage graph for code review analytics: A recovery approach and empirical study,

    T. Hirao, S. McIntosh, A. Ihara, and K. Matsumoto, “The review linkage graph for code review analytics: A recovery approach and empirical study,” inProceed- ings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the JOURNAL OF LATE...

    2019

  15. [23]

    Defining software ecosystems: a survey of software platforms and business network governance,

    S. Jansen and M. A. Cusumano, “Defining software ecosystems: a survey of software platforms and business network governance,” inSoftware ecosystems. Edward Elgar Publishing, 2013, pp. 13–28

    2013

  16. [24]

    What causes my test alarm? automatic cause analysis for test alarms in system and integration testing,

    H. Jiang, X. Li, Z. Yang, and J. Xuan, “What causes my test alarm? automatic cause analysis for test alarms in system and integration testing,” in2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 712–723

    2017

  17. [25]

    On wasted contributions: Understanding the dynamics of contributor-abandoned pull requests–a mixed-methods study of 10 large open-source projects,

    S. Khatoonabadi, D. E. Costa, R. Abdalkareem, and E. Shihab, “On wasted contributions: Understanding the dynamics of contributor-abandoned pull requests–a mixed-methods study of 10 large open-source projects,” ACM Transactions on Software Engineering and Method- ology, vol. 32...

    2023

  18. [26]

    Struc- ture and evolution of package dependency networks,

    R. Kikas, G. Gousios, M. Dumas, and D. Pfahl, “Struc- ture and evolution of package dependency networks,” in2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 2017, pp. 102–112

    2017

  19. [27]

    Parallel sets: Interactive exploration and visual analysis of categorical data,

    R. Kosara, F. Bendix, and H. Hauser, “Parallel sets: Interactive exploration and visual analysis of categorical data,”IEEE transactions on visualization and computer graphics, vol. 12, no. 4, pp. 558–568, 2006

    2006

  20. [28]

    Root causing flaky tests in a large-scale industrial setting,

    W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thum- malapenta, “Root causing flaky tests in a large-scale industrial setting,” inProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 101–111

    2019

  21. [29]

    A study on the lifecycle of flaky tests,

    W. Lam, K. Mus ¸lu, H. Sajnani, and S. Thummalapenta, “A study on the lifecycle of flaky tests,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1471–1482

    2020

  22. [30]

    idflakies: A framework for detecting and partially clas- sifying flaky tests,

    W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “idflakies: A framework for detecting and partially clas- sifying flaky tests,” in2019 12th ieee conference on software testing, validation and verification (icst). IEEE, 2019, pp. 312–322

    2019

  23. [31]

    When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla,

    J. Lampel, S. Just, S. Apel, and A. Zeller, “When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Con- ference and Symposium on the Foundations of Software Engi...

    2021

  24. [32]

    Cost of flaky tests in con- tinuous integration: An industrial case study,

    F. Leinen, D. Elsner, A. Pretschner, A. Stahlbauer, M. Sailer, and E. J ¨urgens, “Cost of flaky tests in con- tinuous integration: An industrial case study,” in2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2024, pp. 329–340

    2024

  25. [33]

    An empirical analysis of flaky tests,

    Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” inProceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, 2014, pp. 643–653

    2014

  26. [34]

    Repeated builds during code review: An empirical study of the openstack community,

    R. Maipradit, D. Wang, P. Thongtanunam, R. G. Kula, Y . Kamei, and S. McIntosh, “Repeated builds during code review: An empirical study of the openstack community,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 153–165

    2023

  27. [35]

    Software ecosystems– a systematic literature review,

    K. Manikas and K. M. Hansen, “Software ecosystems– a systematic literature review,”Journal of Systems and Software, vol. 86, no. 5, pp. 1294–1306, 2013

    2013

  28. [36]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The annals of mathematical statistics, pp. 50–60, 1947

    1947

  29. [37]

    An empirical study of the impact of modern code review practices on software quality,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,”Empirical Software Engineering, vol. 21, pp. 2146–2189, 2016

    2016

  30. [38]

    Towards an interdis- ciplinary, socio-technical analysis of software ecosystem health,

    T. Mens, B. Adams, and J. Marsan, “Towards an interdis- ciplinary, socio-technical analysis of software ecosystem health,”arXiv preprint arXiv:1711.04532, 2017

    2017 arXiv

  31. [39]

    Analysing the evolution of social aspects of open source software ecosystems

    T. Mens and M. Goeminne, “Analysing the evolution of social aspects of open source software ecosystems.” in IWSECO@ ICSOB, 2011, pp. 1–14

    2011

  32. [40]

    The state of continuous integration testing@ google,

    J. Micco, “The state of continuous integration testing@ google,” inICST, 2017

    2017

  33. [41]

    Flaky tests at google and how we mitigate them,

    Micco, John, “Flaky tests at google and how we mitigate them,” https://testing.googleblog.com/2016/05/ flaky-tests-at-google-and-how-we.html, 2016, [Accessed 17-02-2025]

    2016

  34. [42]

    Morville and L

    P. Morville and L. Rosenfeld,Information architecture for the World Wide Web: Designing large-scale web sites. O’Reilly Media, 2006

    2006

  35. [43]

    Experience re- port: An empirical study of api failures in openstack cloud environments,

    P. Musavi, B. Adams, and F. Khomh, “Experience re- port: An empirical study of api failures in openstack cloud environments,” in2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2016, pp. 424–434

    2016

  36. [44]

    D. Neary. (2015, 3) Upstream first: Turning openstack into an nfv platform. Accessed: 2024- 06-03. [Online]. Available: https://www.redhat.com/en/ blog/upstream-first-turning-openstack-nfv-platform

    2015

  37. [45]

    Towards language-independent brown build detection,

    D. Olewicki, M. Nayrolles, and B. Adams, “Towards language-independent brown build detection,” inPro- ceedings of the 44th International Conference on Soft- ware Engineering, 2022, pp. 2177–2188

    2022

  38. [46]

    Openstack repositories hosted on opendev gerrit code review,

    OpenDev, “Openstack repositories hosted on opendev gerrit code review,” https://review.opendev.org/admin/ repos/q/filter:openstack/, [Accessed 10-09-2025]

    2025

  39. [47]

    Third party testing policies for opendev,

    ——, “Third party testing policies for opendev,” https://docs.opendev.org/opendev/system-config/latest/ third party.html#requirements, [Accessed 21-05-2025]

    2025

  40. [48]

    Opensearch dashboards,

    OpenSearch, “Opensearch dashboards,” [Accessed 2025-12-11]. [Online]. Available: https://opensearch.logs.openstack.org/ dashboards/app/ login?nextUrl=%2F dashboards%2Fapp%2Fdiscover% 3Fsecurity tenant%3Dglobal

    2025

  41. [49]

    2023-07-24 unmaintained status replaces extended maintenance,

    OpenStack, “2023-07-24 unmaintained status replaces extended maintenance,” https://governance.openstack. org/tc/resolutions/20230724-unmaintained-branches. html, 2025, [Accessed 13-02-2025]

    2023

  42. [50]

    The flaky test servers function,

    ——, “The flaky test servers function,” https://opendev.org/openstack/openstacksdk/src/ JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 16 commit/d5ba0c42a9d2ed489aca3a6e431da59be3cf39c8/ openstack/tests/unit/test stats.py#L235, 2025, [Accessed 13-02-2025]

    2020

  43. [51]

    How to handle test failures,

    ——, “How to handle test failures,” https: //docs.openstack.org/project-team-guide/testing.html# how-to-handle-test-failures, 2025, [Accessed 13-02- 2025]

    2025

  44. [52]

    The most frequent flaky api test,

    ——, “The most frequent flaky api test,” https://opendev.org/openstack/tempest/src/commit/ 357fc7ec2dfb1fb27d52cd518375e4d8ec977733/tempest/ api/compute/servers/test server rescue.py#L230, 2025, [Accessed 13-02-2025]

    2025

  45. [53]

    The most frequent flaky unit test,

    ——, “The most frequent flaky unit test,” https://opendev.org/openstack/nova/src/commit/ 796250fa0d325057868572f8a4bb29aacf1edcf6/nova/ tests/unit/db/main/test migrations.py#L132, 2025, [Accessed 13-02-2025]

    2025

  46. [54]

    Openstack neutron review 920766,

    ——, “Openstack neutron review 920766,” https:// review.opendev.org/c/openstack/neutron/+/920766, 2025, [Accessed 13-02-2025]

    2025

  47. [55]

    Openstack neutron review 921420,

    ——, “Openstack neutron review 921420,” https:// review.opendev.org/c/openstack/neutron/+/921420, 2025, [Accessed 13-02-2025]

    2025

  48. [56]

    Openstack project team guide,

    ——, “Openstack project team guide,” https: //docs.openstack.org/project-team-guide/testing.html, 2025, [Accessed 13-02-2025]

    2025

  49. [57]

    Testing neutron,

    ——, “Testing neutron,” https://docs.openstack. org/neutron/latest/contributor/testing/testing.html# structure-of-the-unit-test-tree, 2025, [Accessed 10-12- 2025]

    2025

  50. [58]

    Full stack testing — neutron 28.0.0.0b2.dev20 documentation,

    OpenStack Neutron, “Full stack testing — neutron 28.0.0.0b2.dev20 documentation,” [Accessed 12-11- 2025]. [Online]. Available: https://docs.openstack.org/ neutron/latest/contributor/testing/fullstack.html

    2025

  51. [59]

    A survey of flaky tests,

    O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1–74, 2021

    2021

  52. [60]

    Impact of continuous integration on code reviews,

    M. M. Rahman and C. K. Roy, “Impact of continuous integration on code reviews,” in2017 IEEE/ACM 14th International Conference on Mining Software Reposito- ries (MSR). IEEE, 2017, pp. 499–502

    2017

  53. [61]

    Github: Reducing flaky builds by 18x,

    Raine, Jordan, “Github: Reducing flaky builds by 18x,” https://github.blog/engineering/engineering-principles/ reducing-flaky-builds-by-18x/, 2020, [Accessed 26-02- 2025]

    2020

  54. [62]

    Bug 1833757,

    Red Hat Bugzilla, “Bug 1833757,” https://bugzilla. redhat.com/show bug.cgi?id=1833757, 2025, [Accessed 13-02-2025]

    2025

  55. [63]

    Riedemann

    M. Riedemann. (2018, 4) Seeding the future of extended maintenance in openstack. Accessed: 2024-06-

    2018

  56. [64]

    Available: https://superuser.openinfra.dev/ articles/extended-maintenance-openstack/

    [Online]. Available: https://superuser.openinfra.dev/ articles/extended-maintenance-openstack/

  57. [65]

    Convergent contemporary soft- ware peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary soft- ware peer review practices,” inProceedings of the 2013 9th joint meeting on foundations of software engineering, 2013, pp. 202–212

    2013

  58. [66]

    Understanding broadcast based peer review on open source software projects,

    P. C. Rigby and M.-A. Storey, “Understanding broadcast based peer review on open source software projects,” inProceedings of the 33rd International conference on software engineering, 2011, pp. 541–550

    2011

  59. [67]

    Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices?

    J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine, “Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices?” in Annual Meeting of the Southern Association for Institu-...

    2006

  60. [68]

    stestr user manual,

    stestr, “stestr user manual,” [Accessed 2025-12-11]. [Online]. Available: https://stestr.readthedocs.io/en/latest/ MANUAL.html#configuration

    2025

  61. [69]

    unittest — unit testing framework,

    unittest, “unittest — unit testing framework,” [Accessed 2025-12-11]. [Online]. Available: https://docs.python. org/3/library/unittest.html#test-discovery

    2025

  62. [70]

    An empirical study of bugs in test code,

    A. Vahabzadeh, A. M. Fard, and A. Mesbah, “An empirical study of bugs in test code,” in2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 101–110

    2015

  63. [71]

    An empirical study of bugs in test code,

    ——, “An empirical study of bugs in test code,” in2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 2015, pp. 101–110

    2015

  64. [72]

    Understanding interob- server agreement: the kappa statistic,

    A. J. Viera, J. M. Garrettet al., “Understanding interob- server agreement: the kappa statistic,”Fam med, vol. 37, no. 5, pp. 360–363, 2005

    2005

  65. [73]

    Can we benchmark code review studies? a systematic mapping study of methodology, dataset, and metric,

    D. Wang, Y . Ueda, R. G. Kula, T. Ishio, and K. Mat- sumoto, “Can we benchmark code review studies? a systematic mapping study of methodology, dataset, and metric,”Journal of Systems and Software, vol. 180, p. 111009, 2021

    2021

  66. [74]

    Understanding shared links and their intentions to meet information needs in modern code review: A case study of the openstack and qt projects,

    D. Wang, T. Xiao, P. Thongtanunam, R. G. Kula, and K. Matsumoto, “Understanding shared links and their intentions to meet information needs in modern code review: A case study of the openstack and qt projects,” Empirical Software Engineering, vol. 26, pp. 1–32, 2021

    2021

  67. [75]

    Characterizing and mitigat- ing self-admitted technical debt in build systems,

    T. Xiao, D. Wang, S. McIntosh, H. Hata, R. G. Kula, T. Ishio, and K. Matsumoto, “Characterizing and mitigat- ing self-admitted technical debt in build systems,”IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 4214–4228, 2021

    2021

  68. [76]

    Research artifact – cross-project flakiness: A case study of the openstack ecosystem,

    T. Xiao, D. Wang, S. McIntosh, Y . Kamei, and H. Hata, “Research artifact – cross-project flakiness: A case study of the openstack ecosystem,” https://doi.org/10.5281/ zenodo.18476912, 2026, [Accessed 04-02-2026]

    2026

  69. [77]

    R. K. Yin,Case study research: Design and methods. sage, 2009, vol. 5

    2009

  70. [78]

    A study on the interplay between pull request review and continuous integration builds,

    F. Zampetti, G. Bavota, G. Canfora, and M. Di Penta, “A study on the interplay between pull request review and continuous integration builds,” in2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER). IEEE, 2019, pp. 38–48

    2019

  71. [79]

    Empirically revisiting the test independence assumption,

    S. Zhang, D. Jalali, J. Wuttke, K. Mus ¸lu, W. Lam, M. D. Ernst, and D. Notkin, “Empirically revisiting the test independence assumption,” inProceedings of the 2014 International Symposium on Software Testing and Analysis, 2014, p. 385–396

    2014

  72. [80]

    Empirically revisiting the JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 17 test independence assumption,

    S. Zhang, D. Jalali, J. Wuttke, K. Mus ¸lu, W. Lam, M. D. Ernst, and D. Notkin, “Empirically revisiting the JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 17 test independence assumption,” inProceedings of the 2014 International Symposium on Software Testing and ...

    2020

  73. [81]

    Nodepool v3.8.0 configuration,

    Zuul, “Nodepool v3.8.0 configuration,” https: //zuul-ci.org/docs/nodepool/3.8.0/configuration.html# attr-build-log-retention, [Accessed 21-05-2025]

    2025

  74. [82]

    Projects hosted on zuul ci platform,

    ——, “Projects hosted on zuul ci platform,” https:// zuul.opendev.org/t/openstack/projects, [Accessed 11-03- 2025]

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.