{"id":"2fa7a933-80d9-43b4-8c13-d9af6eb1b2a0","arxiv_id":"2602.09311","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Shared CI jobs and test code spread flaky-test failures across most OpenStack projects, and the same test can be flaky in some projects but stable in others.","lead":"Flaky software tests—ones that randomly pass or fail—can break builds in more than one project at the same time. This study measures that 'cross-project flakiness' across 649 OpenStack projects and estimates it wastes years of computing time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-project counts rest on same-name job/test identity without verifying shared definitions or code; name collisions could inflate the 55% and 1,535/1,105 figures.","rationale":"The paper's central contribution is quantifying cross-project flakiness in a large ecosystem. The load-bearing premise is that a recurring job name or test name across projects denotes a shared artifact. Without this, the 55% project-impact figure and the 1,535/1,105 test counts lose their semantic meaning. The paper explicitly asserts that the fully-qualified test name 'ensures tests with this identifier correspond to semantically equivalent code' (Section VI-A) but supplies no check for jobs and no source comparison for tests. This is the weakest link because every headline number depends on it; the data-loss and scope-labeling issues noted in the reader's rationale affect precision and interpretation, but the identity assumption affects whether the phenomenon itself is measured at all. I agree with the reader's weakest assumption. The paper does have credible scaffolding: a replication package, a documented real-world example with two concrete reviews, and a qualitative taxonomy with inter-rater agreement. However, the identity assumption is easily testable, and the verdict should remain CONDITIONAL: if a validation sample shows substantial name collisions, the counts and the 55% figure would need downward revision; if collisions are rare, the central claim stands. I recommend the paper add this validation or soften the claims.","tokens_in":23998,"tokens_out":5807,"duration_ms":50498,"concrete_test":"Sample 50 of the 371 cross-project flaky jobs and 100 of the 1,535 cross-project flaky tests. For each sampled job, inspect the Zuul job definitions in each involved project's repository (e.g., zuul.d/ directory) and determine whether they inherit from a common parent or are identical. For each sampled test, resolve the fully-qualified name to the source file in each project's repo (via the replication package's raw data or Gerrit) and compare the test method bodies (e.g., normalized-source diff/hash). If definitions or bodies differ for more than 5% of sampled pairs, the cross-project classification is partly an artifact of name reuse and the headline counts need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core prevalence claims (Observation 1: 55% of projects; Observation 4: 1,535 cross-project and 1,105 inconsistent flaky tests) rest entirely on the identity-by-name assumption. Section V-A identifies cross-project flakiness by checking whether the same Zuul job name recurs across projects, and Section VI-A treats fully-qualified class+method names as guaranteeing 'semantically equivalent code' (Section VI-A, second paragraph). No verification is reported that same-named jobs actually inherit a common definition or that same-named tests correspond to the same source code. In OpenStack, a project can define a job with the same name as another project's job but with a different parent, parameters, or locally overridden body; likewise, tests with identical fully-qualified names can exist in forks, vendored copies, or independently maintained code. If such collisions are non-negligible, the 378 affected projects and the 1,535/1,105 counts overstate genuine propagation. The paper's construct-validity discussion (Section X) only mentions misclassification from insufficient collection, not from name collisions, so this assumption is unaddressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of flaky tests in the OpenStack ecosystem, focusing on two new phenomena: cross-project flakiness (the same flaky test or CI job affecting multiple projects) and inconsistent flakiness (a test flaky in some projects but stable in others). The authors collect one year of Gerrit/Zuul CI data (June 2023–June 2024), identify 29,911 flaky builds from repeated builds on the same patch set, and then drill down to test-level results for a subset of those builds. They report that cross-project flakiness affects 55% of OpenStack projects, identify 1,535 cross-project and 1,105 inconsistently flaky tests, find that 70% of unit tests exhibit cross-project flakiness, and qualitatively attribute inconsistent flakiness to event-related, dependency-related, and configuration-related causes. They also quantify extra review delay and gather developer feedback through a questionnaire. A replication package is provided.","tokens_in":24276,"tokens_out":6705,"duration_ms":55373,"significance":"If the headline claims hold, this is one of the first ecosystem-level studies of flakiness and would be a useful contribution to the flaky-test and CI literature. The paper has real strengths: a public replication package, test-level analysis in addition to build-level analysis, a manually coded qualitative dataset with reported inter-rater agreement (Fleiss kappa = 0.726), statistical tests with effect sizes, and a small external validation on the Zuul ecosystem. However, the quantitative claims currently outrun the evidence in two ways: the identity-by-name assumption for jobs and tests is asserted rather than verified, and the headline test counts are derived from a heavily filtered subset without making that clear in the abstract. The core existence claim—that flaky tests and jobs recur across project boundaries in OpenStack—is plausible and supported by the data, but the specific prevalence numbers need substantial validation and re-framing before they can be taken at face value.","major_comments":[{"comment":"The paper operationalizes cross-project flakiness by checking whether the same Zuul job name (Section V-A) or the same fully-qualified test name (Section VI-A) recurs across projects. Section VI-A asserts that this identifier 'ensures tests with this identifier correspond to semantically equivalent code,' but no verification is reported. In OpenStack, two projects can define jobs with the same name but different parents, parameters, or local bodies, and the same fully-qualified test name can appear in forks or vendored copies. Section X only discusses misclassification due to insufficient collection, not name collisions. This is load-bearing because Observation 1 (378 projects, 55%) and Observation 4 (1,535/1,105 tests) are the paper's headline results. Please either verify shared definitions via Zuul config parent chains and test source identity, quantify the collision rate, or soften t","section":"V-A, VI-A, X"},{"comment":"Observation 4's counts of 1,535 cross-project and 1,105 inconsistent flaky tests are not counts of all flaky tests. They are sums from the bottom panel of Table II, which is based on the 2,403 tests that remain after excluding single-patch-set tests (Section VI-B). The top panel of Table II counts 11,506 flaky tests, so the two panels use different denominators. The abstract presents 1,535/1,105 as global totals, and Observation 5's '70% of unit tests' is likewise conditional on the filtered set. This is misleading. Please report both raw and filtered counts in the abstract and results, or explain why the filter is appropriate for the headline claim.","section":"VI-B, Table II, Abstract"},{"comment":"Observation 2 reports a growth from 52 affected projects in 2020 to 378 in 2024, but Section IV-B states that data collection covers June 2023–June 2024. It is not explained how 2020–2022 data were obtained. If the one-year dataset was used to construct the time series, the pre-2023 portion is unsupported; if data from a prior study were merged, that source and the different collection methods need to be described. This result underpins the conclusion that cross-project flakiness has 'rapidly accumulated over time' and requires either proper longitudinal data or removal.","section":"V-B, Observation 2"},{"comment":"The test-level analysis is based on 3,369 of 29,911 flaky builds (11%), spanning 370 of 1,651 unique jobs. The paper does not analyze whether the retrievable subset differs systematically from the non-retrievable subset (e.g., by project, job, or retention policy). If log retention is not independent of flakiness, the 1,535/1,105 counts and Observation 5's scope percentages may be biased. Please compare retrievable versus non-retrievable builds/jobs/projects on available observables and discuss the selection bias risk.","section":"VI-A"}],"minor_comments":[{"comment":"Typo: 'prorogation' should be 'propagation'.","section":"IV-A"},{"comment":"The sentence 'This filtering yields 2,403 flaky tests from 11,506 flaky instances' is confusing; it should likely read 'from 11,506 flaky tests' or 'from 57,124 flaky instances.'","section":"VI-B"},{"comment":"The left and right axes are not labeled. The left axis shows frequency of flaky builds and the right axis shows number of projects, but this is not stated in the caption.","section":"Figure 2"},{"comment":"The top (test scope) and bottom (flakiness level) panels use different test populations (11,506 vs 2,403). Add a note explaining the difference and which observations rely on which panel.","section":"Table II"},{"comment":"Observation 8 refers to 'additional build execution time,' but the baseline for 'additional' is not explicitly defined. Please state precisely how the time waste for a flaky build is measured.","section":"VIII"},{"comment":"The external-validation claim is based on only six Zuul projects, one cross-project job, and three inconsistent flaky tests. The phrase 'confirm the applicability of our results in Zuul' is stronger than this evidence supports; please temper the wording.","section":"X"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a good replication package, but the central prevalence numbers currently rest on an unverified naming-identity assumption and a filtered subset that is not consistently labeled. These issues are fixable with additional validation and revision, so I recommend major revision rather than rejection. The temporal claim in Observation 2 also needs clarification, as it appears inconsistent with the one-year data collection window."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe two things to know: cross-project flakiness as a named phenomenon is real and worth studying, and the paper's existence proof is solid — the Cinder/Glance example is compelling. But the 55% project figure and the 1,535/1,105 test counts rest on the assumption that identical job and test names imply shared artifacts, and the paper doesn't verify that. Treat those numbers as upper bounds, not measurements.\n\nWhat's genuinely new: prior flakiness work is single-project; this defines cross-project and inconsistent flakiness at test level, and shows they behave differently across test scopes. That's a useful frame. The paper is also honest about data loss (only 3,369 of 29,911 flaky builds yield test results) and provides a replication package. The qualitative taxonomy is built from 110 coded tests with a reported kappa of 0.726, which is acceptable for an exploratory case study.\n\nThe soft spots, in order:\n\n- Identity-by-name. The stress-test concern is right. A Zuul job named 'tempest-integrated-storage' may not share a definition across projects; a fully-qualified test name can collide across forks or vendored code. The paper asserts the name 'ensures' semantic equivalence, which is too strong. This doesn't kill the result — the real example shows genuine propagation — but it could easily halve or inflate the prevalence. The authors should sample-check the shared definitions and report collision rates.\n\n- The growth-curve claim. I can't reconcile a 2020 point with a stated June 2023–June 2024 collection window. Either they used historical Zuul/comment data not described in Section IV, or the figure is mislabeled. Needs clarification.\n\n- The 70% unit-test surprise is brittle. It depends on keyword-based scope classification and the single-patch-set filter that discards 79% of tests. The paper acknowledges the classification is approximate, but the 'surprise' is presented as stronger than the evidence supports. Also, OpenStack 'unit' tests are not necessarily isolated.\n\n- Time-waste effect size is small (Cliff's delta small), so 'significantly increases' is statistically true but practically modest.\n\nOverall, the central phenomenon holds; the magnitudes don't yet. I'd accept it for peer review and ask for the identity validation and the growth-data clarification. It deserves a serious referee and would be a reasonable reading-group pick for the empirical-SE crowd.\n\nRecommendation: engage with it, but push for the shared-artifact audit.","headline":"A credible first look at ecosystem-level flakiness whose headline magnitudes need validation of the shared-name assumption before I'd cite the numbers.","tokens_in":24792,"tokens_out":4218,"would_cite":true,"duration_ms":36855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that flaky tests routinely spread across project boundaries in the OpenStack ecosystem, affecting 55% of its Zuul-based projects and measurably slowing code review.","keywords":["flaky tests","cross-project flakiness","OpenStack","continuous integration","Zuul CI","code review","test scopes","inconsistent flakiness"],"falsifier":"Pick the 1,535 cross-project flaky tests, fetch the source of each named test in each project, and compare code plus Zuul job definitions. If a substantial share (say, more than 10%) of same-named tests turn out to be different implementations or locally overridden jobs, the cross-project interpretation collapses to a naming artifact. A cheaper first check: inspect the inheritance graph of the 371 cross-project flaky jobs to confirm each job's definition is literally shared, not coincidentally named.","tokens_in":1347,"feed_emoji":"🧪","tokens_out":1416,"duration_ms":43396,"temperature":0.7,"pith_summary":"The paper tries to establish that test flakiness is not just a per-project problem: in a large interconnected CI ecosystem, a single non-deterministic test can fail in many projects at once. Analyzing a year of OpenStack code-review and CI data, it reports 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests, with 55% of Zuul-using projects touched. It argues that cross-project spread is more frequent and more costly than single-project flakiness, and that even unit tests—supposedly isolated—appear in 70% of cross-project cases. A sympathetic reader would care because ecosystem-level flakiness shifts where mitigation effort belongs: shared CI jobs and standardized configurations, not just individual test code.","feed_headline":"Flaky tests hit 55% of Zuul-run OpenStack projects","feed_subtitle":"A year of CI logs reveals 1,535 shared flaky tests—unit tests included.","key_machinery":"The load-bearing object is the shared CI artifact identity: a Zuul job name (coarse-grained) or a fully-qualified test name (fine-grained) that recurs across multiple OpenStack projects. The paper treats that recurrence as evidence that the same job definition or test code is shared through inheritance (for example, the devstack-tempest parent job), then labels the recurrence cross-project flakiness. Flakiness levels are built from the matrix of which projects run a given test and which ones see it fail: Shared-Partial Trigger, Uniform Multi-Project, Cross-Trigger, and Project-Isolated. This classification carries the prevalence, cost, and cause arguments.","core_discovery":"The central discovery is that flaky tests in OpenStack frequently operate across project boundaries: same-named tests and shared CI jobs fail non-deterministically in multiple projects, not just one. Combining rerun detection in code-review comments with build-log analysis over June 2023–June 2024, the authors identify 29,911 flaky builds, 1,535 cross-project flaky tests, and 1,105 inconsistently flaky tests; 378 projects (about 55% of the 691 that use Zuul CI) are affected. Cross-project flaky builds waste more review time than project-isolated ones (significant at p<0.001), and 70% of flaky unit tests show uniform multi-project behavior, contradicting the assumption that unit tests are ins","pith_inferences":["If the same-name identity assumption holds only weakly, the true rate of shared flakiness may be lower; a replication that verifies job inheritance and test-code equality across projects would sharpen or shrink the estimate. This is an inference about the paper's load-bearing premise, not a paper claim.","The four-level flakiness taxonomy could be reused as a monitoring signal: a test shifting from Project-Isolated to Shared-Partial Trigger would indicate a shared resource or environment degrading. The paper does not propose this use.","Since 89% of inconsistent flakiness is event-related, many 'flaky tests' may really be flaky environments; if so, ephemeral containerized CI environments with pinned resources could remove a large share of cross-project flakiness without touching test code. The paper suggests environment standardization but does not test it.","The paper's time-waste result (1,156 days across all flaky builds) implies that ecosystem-level CI coordination has direct economic consequences; extending the same measurement to other ecosystems would reveal whether OpenStack is typical or extreme."],"forward_implications":["If flaky tests propagate across projects, fixing them becomes a coordination problem: one shared job failure can block reviews in many projects at once.","Standardizing CI configurations and pinning dependencies across the ecosystem could remove a large share of inconsistent flakiness, since configuration and dependency mismatches are among the top identified causes.","The 'recheck and wait' paradigm amplifies waste; replacing it with early intervention on flaky tests and flaky jobs would save review time and CI resources.","Unit tests being cross-project flaky implies that test isolation, mocking, and dependency injection need rethinking even for unit tests reused across projects.","Other large CI ecosystems can use the 55% figure and the 1,535/1,105 counts as a baseline when measuring their own cross-project flakiness."],"fun_headline_variants":["Flaky tests leap across OpenStack projects","70% of flaky unit tests span projects","Cross-project flakiness wastes review time","1,535 flaky tests cross project boundaries","Unit tests flake across OpenStack too"],"cache_read_input_tokens":26112,"weakest_assumption_plain":"The counts of cross-project flakiness assume that two projects running the same Zuul job name or the same fully-qualified test name are running the same shared job definition and the same test code; if those names merely collide across forks or locally overridden definitions, the 55% figure and the 1,535/1,105 counts include artifacts of naming.","fun_headline_variants_meta":{"raw":{"variants":["Flaky tests leap across OpenStack projects","70% of flaky unit tests span projects","Cross-project flakiness wastes review time","1,535 flaky tests cross project boundaries","Unit tests flake across OpenStack too"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1687,"prompt_tokens":815,"completion_tokens":872,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":804}},"tokens_in":559,"tokens_out":872,"duration_ms":7932,"temperature":1.0,"reasoning_tokens":804,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T02:58:18.521701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick the 1,535 cross-project flaky tests, fetch the source of each named test in each project, and compare code plus Zuul job definitions. If a substantial share (say, more than 10%) of same-named tests turn out to be different implementations or locally overridden jobs, the cross-project interpretation collapses to a naming artifact. A cheaper first check: inspect the inheritance graph of the 371 cross-project flaky jobs to confirm each job's definition is literally shared, not coincidentally named.","supporting_citations":[],"review_version":1}