{"id":"fbafd7c8-eb91-45ee-aa80-709933589454","arxiv_id":"2501.04976","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An industrial study that builds a 46-category taxonomy of flaky job failures and uses RFM clustering to select 14 priority categories for automated diagnosis.","lead":"This paper analyzes 4,511 flaky job failures from TELUS's CI/CD pipelines and organizes them into 46 failure categories. It then ranks these categories by recency, frequency, and estimated cost, highlighting 14 priority targets for automated diagnosis and repair.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing-log window, not the rerun heuristic, is the load-bearing threat: 33% of flaky failures are discarded and the gap coincides with the temporal/recency claims.","rationale":"The reader identifies the rerun-based flaky heuristic as the weakest assumption. I read that heuristic as standard and acceptable for an industrial CI study: the paper explicitly defines flaky job failure as a non-deterministic outcome on rerun, and the heuristic operationalizes that definition with interview confirmation. A more serious threat is that the paper discards 33.29% of flaky failures because logs are missing, then another 14.2% because the tool cannot label them, leaving only 58.1% of the originally detected flaky failures. The missingness has a known mechanism—a GitLab migration in early 2022—that is exactly the period in which RQ3 detects a 'big blank period' and around which RQ4's recency claims are built. Since R, F, and M are all computed from the 4,511 labeled failures, the priority list in Table VI can be biased by any category whose occurrences fell predominantly in the missing window. The paper acknowledges this limitation but does not test whether the priority set is stable under reasonable missing-data assumptions. This is a conditional-accept issue, not a rejection: the taxonomy of categories is likely real, and the top categories (e.g., misconfigured env variable) are so frequent that additional missing data would probably not dislodge them, but the completeness of the 14-priority list and the 'emerging/idle' labels are not yet supported. A sensitivity analysis or metadata-based reconstruction would settle the concern. I therefore keep the reader's CONDITIONAL verdict and partially agree with their characterization: the dataset-selection concern is right, but the load-bearing piece is the non-random log-missing gap rather than the rerun heuristic itself.","tokens_in":16354,"tokens_out":11336,"duration_ms":120571,"concrete_test":"Re-estimate RQ3/RQ4 after applying the labeling tool to all 7,763 flaky failures, using a coarse category inferred from structured GitLab job metadata (failure_reason, exit code, runner tags, duration, timestamp) for the 2,507 missing-log jobs; if the top-14 priority set changes, the missing-log exclusion is not neutral and the central prioritization is not robust. Alternatively, if metadata are insufficient, recompute the RFM clusters using only post-Q1-2023 failures (after the blank period) and compare the priority set with Table VI; divergence would demonstrate that recency/frequency conclusions are driven by the gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B excludes 2,507/7,763 (33.29%) flaky failures because logs are missing, and another 745 remain unlabeled; the final analysis uses 4,511/7,763 (58.1%). The missing logs are not random: they follow a GitLab migration in early 2022, which the paper itself identifies as the 'big blank period' in Fig. 4. RQ3 and RQ4 compute Recency from observed occurrence dates and frequency from the labeled set, so any category that occurred mainly inside the blank window is artificially old and undercounted. The paper's statement that 'docker daemon connection failure...have not occurred for over a year' and that 'api gateway deployment error' emerged only recently is indistinguishable from 'their logs were lost during this period.' Because R, F, and M are all computed from the 4,511 labeled failures, the priority list in Table VI inherits this non-random missingness. The paper acknowledges the limitation but provides no sensitivity analysis to show the 14 priority categories survive alternative assumptions about the missing window.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of 4,511 flaky job failures from 80 TELUS projects over a 6.5-year period. The authors develop a semi-automated regex-based labeling tool, classify failures into 46 categories, estimate each category's recency, frequency, and monetary cost, and then apply k-means clustering on RFM scores to identify 14 priority categories for automated diagnosis and repair. The paper's contributions are a taxonomy of flaky job failure categories and an RFM-based prioritization framework that the authors propose as transferable to other contexts.","tokens_in":16520,"tokens_out":5127,"duration_ms":47892,"significance":"If the taxonomy and priority list are reliable, the study fills a real gap between flaky test classification and flaky job detection by providing a diagnosis-oriented taxonomy and a concrete prioritization method. The scale of the industrial dataset, the involvement of TELUS engineers in validating labels, and the promise of a replication package are notable strengths. However, the validity of the central claims depends on two currently weak points: the non-random exclusion of missing logs, which overlaps with the temporal window used for recency claims, and the non-independent evaluation of the labeling tool. The RFM prioritization also relies on several unvalidated parameter choices. These issues do not invalidate the taxonomy as a qualitative contribution but they do undermine the specific priority ranking unless addressed with sensitivity analyses.","major_comments":[{"comment":"The exclusion of 2,507 flaky failures with missing logs (33.29%) is not random: the missing period coincides with the 'big blank period' between early 2022 and Q1 2023, as shown in Figure 4. Because Recency and Frequency in RQ3 and RQ4 are computed from the observed labels only, the statement that 'docker daemon connection failure ... have not occurred for over a year' and the claim that 'api gateway deployment error' appeared only recently are indistinguishable from log loss during this period. The paper acknowledges this limitation in Section V but provides no sensitivity analysis. I request a robustness analysis that shows whether the 14 priority categories in Table VI survive alternative treatments of the missing window, for example by restricting the analysis to the post-migration period, by imputing missing occurrences, or by recomputing RFM scores after removing the entire blank period.","section":"Section III-B, RQ3/RQ4 (Fig. 4, Tables IV and VI)"},{"comment":"The reported recall of 91.34% on Sample 2 is not an independent estimate of labeling quality because the tool was iteratively refined using the unlabeled portion of Sample 2 until recall exceeded 90%; the final value is an in-sample measure. In addition, the text states 'Our labeling tool achieves a precision score of 91% on an unseen representative sample,' but Table II lists precision 99.31% and recall 91.34% at the final iteration, and Section V refers to '≈82% precision' where Table II shows 81.56% recall. Since the correctness of the taxonomy depends on label quality, the authors should either evaluate on a fresh held-out sample that is entirely uninvolved in tool development, or report the iteration-0 recall as the honest out-of-sample estimate and clearly distinguish it from the final recall.","section":"Section III-C, Table II"},{"comment":"The prioritization depends on several user-chosen parameters whose influence is not examined: the number of clusters k=8, the IsolationForest contamination ratio of 10%, the choice to average recency over the last three occurrences, and the quintile-based RFM scoring. Because the paper's central contribution is the priority list, I ask for a stability analysis over a plausible grid of these parameters, or a bootstrap resampling across categories, to show that the set of 14 priority categories is not an artifact of one particular parameterization.","section":"Section IV-D, Tables V and VI"},{"comment":"The diagnosis cost is estimated as the time from the initial failure to the last rerun in the same rerun sequence, which may include waiting periods, weekends, and unrelated delays; the paper acknowledges this in Section V. However, Figure 3 shows that diagnosis cost constitutes roughly 99% or more of every category's total cost, so the monetary (M) component in the RFM model is effectively determined by this proxy. Any systematic bias in the proxy, such as categories whose rerun sequences are naturally longer (e.g., job execution timeout), will propagate directly into the priority rankings. I recommend a sensitivity analysis that caps the delay, excludes non-working hours, or uses an alternative cost measure, to confirm that the top priority categories remain unchanged.","section":"Section IV-B, Eq. (2) and Fig. 3"}],"minor_comments":[{"comment":"The sentence 'we developed a regex-based labeling tool that achieved ≈82% precision on an unseen sample' should refer to recall, not precision; Table II reports 99.31% precision and 81.56% recall at iteration 0.","section":"Section V, Internal Validity"},{"comment":"The meaning of the 'Std' column is unclear; the header appears to combine the standard deviation of RFM scores with the average values in an ambiguous way, and the text should explain what this number represents.","section":"Table V"},{"comment":"The flakiness heuristic requires a rerun on the same commit with an outcome change; this systematically excludes flaky failures that are never rerun or whose reruns occur on different commits. The paper notes this aligns with TELUS practices, but the possible impact on the category distribution should be explicitly acknowledged in the limitations.","section":"Section III-B"},{"comment":"The phrase 'a precision of 100% and a recall of 99.44%' for Sample 1 is clear, but the later sentence 'Our labeling tool achieves a precision score of 91% on an unseen representative sample' is a misstatement of the reported metrics and should be corrected.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"This is a solid industrial case study with a plausible taxonomy, but the missing-log gap and the non-independent label validation are load-bearing for the priority list. The authors should be encouraged to add the requested sensitivity analyses; without them, the concrete RFM rankings should be treated as illustrative rather than definitive. The single-company scope is acceptable for an empirical SE study, but the framing may need to be adjusted in the conclusion to avoid overgeneralizing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first empirical taxonomy of flaky job failure causes (not just flaky tests) built from a large industrial dataset, and the RFM prioritization is a simple but sensible way to turn that taxonomy into an action list. I think the core contribution—46 categories with frequencies and project spread—is real and will be useful to CI researchers and practitioners. The paper is worth serious referee time.\n\nWhere it gets soft: the missing-log problem is not a side worry, it's load-bearing. 33% of flaky failures have no logs, and the blank window is exactly the period around the 2022 GitLab migration. RQ3's claims that docker daemon connection failure has vanished and api gateway deployment error only recently appeared are indistinguishable from 'logs for those failures were lost.' R, F, and M are all computed from the labeled 4,511 failures, so Table VI inherits that non-random missingness. The paper acknowledges the limitation in Threats to Validity but offers no sensitivity analysis. That needs to be addressed before the priority list can be taken as stable.\n\nSecond soft spot: the labeling tool's reported recall is not an independent estimate. Sample 2 was used to refine the tool and then used to evaluate it; the 91.34% is therefore optimistic. A truly held-out sample is needed. Also, the numbers in Section V appear scrambled: it says precision ~82% on an unseen sample, but the reported precision on Sample 2 is 99.31% and the recall is 81.56%. Those should be reconciled.\n\nThird, the cost model is coarse—lead time as a proxy for diagnosis time, flat salary rate, no sensitivity analysis—but this is less damaging because the prioritization is mostly driven by frequency and recency, and the authors say cost is for comparative purposes. Still, a simple one-way sensitivity on S and M would help.\n\nThe rerun heuristic the reviewer worried about is actually fine: defining flaky as outcome changes on same-commit reruns is standard and acknowledged. I don't see a circularity problem in the taxonomy itself.\n\nBottom line: the taxonomy and the general approach are solid and new. The current version overstates confidence in the specific priority rankings because of missing data and the evaluation design. This deserves peer review, but it needs revision: held-out evaluation, sensitivity analysis for the missing window, and cleaned-up accuracy numbers.\n\nI'd bring it to reading group and would cite it for the taxonomy. Recommendation: send to review, with a request for major revision.","headline":"Useful first taxonomy of flaky job failure causes, but the priority list is built on non-random missing logs and an optimistic tool evaluation.","tokens_in":17114,"tokens_out":2182,"would_cite":true,"duration_ms":19904,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flaky job failures fall into 46 identifiable categories, with 14 priority targets for automated diagnosis and repair.","keywords":["flaky jobs","flaky job failures","failure categories","RFM analysis","prioritization","continuous integration","continuous deployment","machine learning"],"falsifier":"Re-run the labeling and RFM ranking after restoring the missing logs from the platform-migration period, or apply the same pipeline to a second organization's CI logs with complete rerun histories; if misconfigured environment variables and job execution timeouts no longer lead the recency-frequency-cost ranking, the priority list depends on the missing data or on this company's specific environment rather than on the categories themselves.","tokens_in":16105,"feed_emoji":"🔁","tokens_out":10542,"duration_ms":92121,"temperature":0.7,"pith_summary":"Flaky job failures—non-deterministic failures that pass or fail on rerun without code changes—waste time and machine resources in continuous-deployment pipelines. The paper analyzes 4,511 labeled flaky job failures from the CI/CD system of a large telecommunications company and claims these failures fall into 46 identifiable categories, with the five most frequent covering more than 40% of all cases. It then ranks the categories by recency, frequency, and estimated monetary cost, and claims that 14 of them are priority targets for automated diagnosis and repair, led by misconfigured environment variables and job execution timeouts. The point of the ranking is to focus future tooling and repair efforts on the failures that are most recent, most recurrent, and most expensive, rather than trying to handle every possible cause.","feed_headline":"14 flaky failure categories emerge as top repair targets","feed_subtitle":"A study of 4,511 job failures puts misconfigured environment variables and timeouts at the head of the list.","key_machinery":"The machinery has three parts. First, a semi-automated labeling tool built from 51 regex search rules assigns each failure log to one of 46 category labels, with categories consolidated by engineers and validated on held-out samples. Second, an RFM model computes, for each category, recency (average days since the last three occurrences), frequency (total occurrence count), and monetary value (estimated machine run cost plus engineer diagnosis delay cost derived from a cost model). Third, these three measures are normalized into RFM scores and clustered with K-means, after five outlier categories are removed with an isolation-forest detector, producing eight clusters that are then interpreted as priority levels from high priority to irrelevant.","core_discovery":"The paper's central claim is that flaky job failures are not an unmanageable mix of one-off causes: they can be labeled into a stable taxonomy of 46 categories, and the categories can be prioritized for automated diagnosis and repair using recency-frequency-monetary (RFM) analysis. The most frequent category, misconfigured environment variables, accounts for about 15% of the labeled failures; the five most frequent categories together account for more than 40%. The five most costly categories overlap only partially with the five most frequent ones, showing that frequency alone is not a sufficient guide. When recency, frequency, and cost are considered together, 14 categories emerge as priorities, with misconfigured environment variables and job execution timeouts at the top. The paper also claims that the RFM-based prioritization approach itself is transferable to other contexts where failure categories must be ranked.","pith_inferences":["One consequence the authors leave implicit is that the same RFM pipeline could be applied to another organization's CI logs; if the leading categories differ sharply, the priority list is likely specific to this company's toolchain rather than a general property of CI flakiness.","The top category also exposes a blind spot the paper notes but does not pursue: environment-variable changes are not version-controlled, so standard 'no code changes' flakiness definitions can miss an entire cause class; versioning or auditing CI environment variables would be a concrete next repair strategy.","Because roughly 30% of flaky failures had missing logs around a major CI-platform migration, the recency and frequency estimates may under-represent categories active during that window; recovering those logs could move some categories in or out of the priority 14."],"forward_implications":["Automated diagnosis and repair work should be aimed first at the 14 priority categories, with misconfigured environment variables and job execution timeouts as the top two targets.","Recency, frequency, and monetary cost should be treated as independent signals: the paper shows that the most frequent categories are not the most costly, so prioritization needs all three measures.","Fourteen additional categories, including idle and emerging issues, should be monitored because they are either recently active but cheap or historically costly and possibly resurgent.","The remaining categories, about 39% of the taxonomy, can be deprioritized as relic, negligible, or irrelevant, freeing effort for the priority set.","A regex-based labeling tool built from manually labeled samples labels 85.83% of the original 5,256 flaky failures automatically, which makes the taxonomy usable beyond the initial study."],"supporting_citations":[{"why":"Supplies the rerun-based flakiness heuristic that selects the jobs studied.","marker":"[7]"},{"why":"Provides the RFM analysis method adapted for prioritizing failure categories.","marker":"[15]"},{"why":"Inspires the cost model that estimates machine and diagnosis costs for each category.","marker":"[22]"},{"why":"Justifies the statistically significant samples used to build and evaluate the labeling tool.","marker":"[17], [18]"},{"why":"Earlier flaky-test taxonomy that motivates categorizing root causes and supplies the flaky test category.","marker":"[4]"},{"why":"Provides the K-means clustering algorithm used to group RFM scores into priority clusters.","marker":"[23]"},{"why":"Supplies the isolation-forest outlier detector used to separate five outlier categories before clustering.","marker":"[25]"}],"fun_headline_variants":["Flaky job failures: 46 categories, 14 top repair targets","RFM analysis ranks flaky failure categories for repair","4,511 flaky failures yield 14 priority categories","Flaky failure causes: env vars and timeouts top list"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a job failure counts as flaky only when re-running the same commit changes the outcome, so failures that are never rerun or whose reruns never alternate success and failure are invisible to the dataset and to every category and priority derived from it.","fun_headline_variants_meta":{"raw":{"variants":["Flaky job failures: 46 categories, 14 top repair targets","RFM analysis ranks flaky failure categories for repair","4,511 flaky failures yield 14 priority categories","Flaky failure causes: env vars and timeouts top list"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1618,"prompt_tokens":940,"completion_tokens":678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":608}},"tokens_in":556,"tokens_out":678,"duration_ms":6381,"temperature":1.0,"reasoning_tokens":608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:28.437672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the labeling and RFM ranking after restoring the missing logs from the platform-migration period, or apply the same pipeline to a second organization's CI logs with complete rerun histories; if misconfigured environment variables and job execution timeouts no longer lead the recency-frequency-cost ranking, the priority list depends on the missing data or on this company's specific environment rather than on the categories themselves.","supporting_citations":[{"cited_title":"Data Mining Using RFM Analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the RFM analysis method adapted for prioritizing failure categories."},{"cited_title":"The Art of Testing Less without Sacrificing Quality,","cited_arxiv_id":null,"evidence_quote":"Inspires the cost model that estimates machine and diagnosis costs for each category."},{"cited_title":"Available: https://scikit-learn/stable/modules/ generated/sklearn.cluster.KMeans.html","cited_arxiv_id":null,"evidence_quote":"Provides the K-means clustering algorithm used to group RFM scores into priority clusters."},{"cited_title":"IsolationForest example","cited_arxiv_id":null,"evidence_quote":"Supplies the isolation-forest outlier detector used to separate five outlier categories before clustering."}],"review_version":1}