{"id":"d56fb4bb-d11a-4999-bf6b-f28ea05c5a3f","arxiv_id":"2504.12517","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"At Meta, over 14% of code changes are explicitly cleanup or reengineering work, and targeted rework cuts authoring time and outage rates substantially.","lead":"Meta engineers spend over 14% of code changes on cleanup and reengineering, far more than the 4% previously reported. A six-month study of rework found large drops in authoring time and outages, though some measures like code centrality moved in unexpected directions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Impact claims rest on uncontrolled pre/post comparisons, and Table VI's reported odds ratios do not follow from the displayed proportions; a control-group re-analysis is needed.","rationale":"The reader's primary weakest assumption was the keyword heuristic behind the 14.2% figure; that is a valid concern, but the more load-bearing issue for the paper's central conclusion is the causal interpretation of the impact results. The impact analysis in Section VII uses pre/post comparisons without control groups for the main outcomes, and the targeted files were selected using those same outcomes as prioritization criteria. This makes regression to the mean and secular trends plausible alternative explanations for the reported gains. Table VI is also inconsistent on its face: the stated odds ratios cannot be derived from the proportions shown, making the outage-reduction magnitudes unverifiable. I would keep the reader's conditional verdict, but the conditions should include supplying raw counts and a control-group analysis for the impact claims. The paper deserves credit for its transparency about context, internal tooling, and prior reengineering literature, and I am not accusing the authors of any misrepresentation; the issue is that the quantitative evidence as presented does not yet support the strength of the causal language in the abstract.","tokens_in":15860,"tokens_out":8955,"duration_ms":93522,"concrete_test":"Ask the authors for the raw pre/post contingency counts behind Table VI and the pre/post diff-level data for H1-H3 and H5. Then (a) recompute odds ratios and exact confidence intervals from the raw counts, and (b) construct a control set of non-reengineered files matched on pre-period outage rate, DAT, and complexity and compare their pre/post changes. If the odds ratios are not 5.2 and 1.55, or if the control files show similar declines, the claimed quality and speed improvements are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VII.A compares outcomes three months before and three months after reengineering for files touched by improvement tasks. For H1 (outages), H2 (DAT), H3 (sessions), and H5 (complexity), there is no comparison or control group; only H4 (centrality) uses matched non-reengineered files. Because the prioritization criteria in Section VI explicitly include prior outages, high DAT, and complexity, the targeted files are selected on the outcome variables, so pre/post improvements can arise from regression to the mean or company-wide trends. The post-intervention set also includes files created, renamed, or deleted during reengineering, with old filenames mapped to current names, so the compared distributions are not necessarily the same populations. Table VI is internally inconsistent as printed: for dead code removal it reports an odds ratio of 5.2 and a \"90% decrease,\" but the displayed proportions (57%/43% no-SEV, 76%/24% SEV) cannot yield OR=5.2 without additional raw denominators, and an OR of 5.2 corresponds to roughly an 81% decrease in odds, not 90%. The same issue applies to the 1.55 OR and \"55% decrease.\" No confidence intervals or raw counts are given for any impact table. The headline claim of substantial improvements in quality and speed is therefore not currently checkable from the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a mixed-methods case study at Meta investigating how a large company counteracts code decay: it catalogues code improvement practices (RQ1), develops prioritization criteria (RQ2), finds organic or 'grass-roots' perfective maintenance (RQ3), and evaluates the impact of reengineering on outages, authoring time, sessions, complexity, and call-graph centrality (RQ4). The authors report that over 14% of diffs are explicitly devoted to code improvement, and that reengineering leads to substantial reductions in outage incidence, DAT (23%–59%), sessions, and complexity, whereas centrality unexpectedly increases. The study combines action research, analysis of internal documents, and an observational before/after analysis of 1,000+ reengineered files.","tokens_in":16086,"tokens_out":7878,"duration_ms":74964,"significance":"If the quantitative claims held, the paper would be one of the few industrial-scale evaluations of code reengineering, with rare insider access to Meta's practices and metrics. Its contributions include a novel catalog of practices (Better Engineering, gamification, badges), a set of operationalized prioritization metrics (Table II), and a replication attempt of earlier reengineering studies [6]–[8]. The paper honestly reports a rejected hypothesis (H4: centrality did not decrease), which is a strength. However, the proprietary data are not released, and the key causal claims rest on observational before/after comparisons without control groups for most outcomes. The significance of the work is therefore conditional on addressing the methodological gaps below.","major_comments":[{"comment":"The pre/post comparisons for H1 (outages), H2 (DAT), H3 (sessions), and H5 (complexity) have no comparison group. Because Section VI describes prioritization criteria that explicitly select files with high DAT, prior outages, and high complexity, the treated files are selected on the outcome variables. The observed improvements could stem from regression to the mean, company-wide trends, or the mapping of old filenames to current names, which changes the composition of the analyzed file set. The matched-control approach used for H4 should be applied to the other outcomes, and the analysis should report the numbers of files and diffs in each period. Without this, the conclusion of 'substantial improvements in quality and speed' is not supported.","section":"Section VII.A, Tables VI–IX"},{"comment":"The reported odds ratios are not reproducible from the displayed proportions. For dead code removal, the text states OR=5.2 and a '90% decrease,' but the SEV proportions (before 76%, after 24%) imply an odds ratio of about 0.1 (a 90% decrease in odds) if interpreted as column percentages, while the no-SEV proportions (57%/43%) imply a different ratio if interpreted as row percentages; no combination of the four displayed numbers yields OR=5.2. An OR of 5.2 would correspond to an 81% decrease in odds (1 – 1/5.2). The same issue applies to the CCN row (OR=1.55 vs '55% decrease'). Please provide the raw 2x2 contingency tables and clarify the direction of the odds ratio.","section":"Section VII.C, Table VI"},{"comment":"The claim that 14.2% of diffs are 'explicitly devoted to code improvement' rests entirely on a keyword heuristic applied to diff titles/tags. The paper reports that 'at least 20 diffs' per keyword were sampled and then 'refined to exclude occasional enhancements,' but no precision, recall, or inter-rater agreement is reported. Without a validation against a reference set of manually classified diffs, the headline figure may over- or under-count perfective maintenance, and the comparison to the prior 4% (from Mockus and Votta [26]) is not credible. At minimum, report the number of sampled diffs, the true/false positive counts per keyword, and a sensitivity analysis with stricter and looser keyword sets.","section":"Section V.A, Table I"},{"comment":"The effect sizes are reported as point estimates without confidence intervals or raw counts. For example, Table VII reports median DAT changes from 0.23 to 0.59 with only a note that a Mann-Whitney test was used; Table IX reports average complexity decreases without a CI. Given the high variance of DAT and complexity metrics, the reader cannot judge the precision of these estimates. Please report 95% confidence intervals (or bootstrap CIs) for each effect size and the number of diffs/files contributing to each comparison.","section":"Tables VII–X"},{"comment":"The internal-validity section does not address the selection-confounding and population-change threats identified above, and it does not account for multiple testing across four task types and five hypotheses. With significance tests run for four task types on each of five outcomes, some significant results are expected by chance; the paper should either apply a correction or explicitly report all tests performed, including non-significant ones.","section":"Section IX.C"}],"minor_comments":[{"comment":"The sentence 'Organic reengineering effort, on the other hand, might be undertaken as part of regular coding activities and might include' ends mid-sentence with no continuation; it should be completed or revised.","section":"Section IX.B"},{"comment":"Grammatical error: 'Our investigation to prioritize... lead to the development' should be 'led to the development.'","section":"Abstract"},{"comment":"The column headers 'Proportion of Diffs (no Sev) during the period' and 'Proportion of SEV-triggering Diffs' are ambiguous; clarify whether the proportions are computed within a period or across periods, and state the denominator for each percentage.","section":"Table VI"},{"comment":"The description of how 'current filename' mapping handles renamed files is difficult to follow; a small illustrative example of the mapping and its effect on the pre/post file sets would improve clarity.","section":"Section VII.A"},{"comment":"The caption notes that internal badges are replaced with GitHub analogues; please also state explicitly that actual badge images are omitted for confidentiality, so readers understand the figure is illustrative rather than literal.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly within the journal's scope as an industrial empirical study. The inconsistent numbers in Table VI and the lack of control groups for most outcomes are the main obstacles; however, they are fixable through re-analysis and a more cautious framing of the causal claims. I would encourage the authors to prioritize those revisions, as the descriptive and qualitative parts of the paper are already valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is worth reading for the descriptive material, but don't trust the impact numbers as causal evidence.\n\nWhat's actually new: the RQ1 finding that 14.2% of diffs are perfective (vs the old 4% from Mockus and Votta), the catalog of practices including Better Engineering's 20-30% allocation guideline and badges, and the use of software-supply-chain centrality to target reengineering. The engineer feedback in Section VI is a nice piece of action research. This is the first detailed public look at how a large rapid-delivery shop handles code decay, and that alone has real value.\n\nWhat it does well: the methods are described honestly, the task taxonomy in Table III is sensible, and the authors are upfront about not being able to release data. The replication of Geppert et al. is a legitimate attempt, and the result that call-graph centrality went up rather than down is reported without spin.\n\nNow the soft spots, in order of importance. First, the impact evaluation (Section VII) compares three months before and after on the same reengineered files, with no control group for H1, H2, H3, H5. Since the prioritization criteria explicitly include prior outages, high DAT, and complexity, these files were selected on the outcomes. Regression to the mean is a live threat, and the abstract's 'substantial improvements' claim overreaches. Second, Table VI is internally inconsistent. For dead code removal, the displayed proportions (57/43 and 76/24) yield an odds ratio around 2.4, not 5.2, and the '90% decrease' doesn't follow from any natural reading. For CCN decompositions, OR=1.55 translates to about a 35% decrease in odds, not 55%. No confidence intervals or raw counts are given. So the outage reductions are not checkable as presented. Third, the keyword heuristic for perfective classification is plausible but unvalidated; the 14% headline depends on it. That's a moderate concern, not fatal.\n\nThe central descriptive claims hold up. The causal claims don't. Who benefits: software engineering researchers and practitioners interested in technical debt and reengineering at scale. I'd bring it to a reading group to discuss design issues. I'd cite it for the 14% figure and the practice catalog, with a caveat. It deserves a serious referee—send it to peer review, but with the expectation of major revision on the impact analysis.","headline":"Descriptive account of Meta's code-improvement practices is genuinely new; the pre/post impact claims are too uncontrolled to support the abstract's causal wording.","tokens_in":16711,"tokens_out":4478,"would_cite":true,"duration_ms":43129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"At Meta, over 14% of code changes are explicitly devoted to code improvement, and reengineering measurably cuts authoring time and outage incidence.","keywords":["code improvement","refactoring","technical debt","dead code removal","Diff Authoring Time","software supply chain","centrality","Better Engineering"],"falsifier":"Take a random sample of diffs at Meta that carry none of the improvement keywords and have engineers label them as perfective or not; if many are perfective despite lacking the keywords, the 14.2% share is a lower bound that could be much higher, while finding that most keyword-tagged diffs are actually feature work would deflate it. A sharper test would apply the same keyword methodology to a codebase where refactoring commits are recorded in a structured field and check whether the classifier recovers that known number.","tokens_in":15680,"feed_emoji":"🧹","tokens_out":4794,"duration_ms":43511,"temperature":0.7,"pith_summary":"The paper aims to show how a large company that ships software rapidly keeps its codebase maintainable by treating code improvement as a continuous, partially structured practice. It reports that at Meta more than 14% of code changes are explicitly devoted to perfective maintenance, a share far above the roughly 4% reported in earlier industry work. It claims that reengineering targeted at dead code, cyclomatic complexity, large classes, and platformization produces measurable gains: authoring time falls, outage-triggering changes fall, and code complexity drops. It also proposes a set of prioritization criteria, centered on a software supply chain network and Katz centrality, for choosing which code to reengineer. A sympathetic reader should take the paper as evidence that deliberate, tool-supported code improvement can coexist with rapid release cycles.","feed_headline":"Code cleanup is 14% of Meta's changes and pays off","feed_subtitle":"Reengineering cuts authoring time by up to 77% and outage-causing diffs by up to 90%.","key_machinery":"The load-bearing machinery is a keyword-based diff classifier plus a before-and-after matched-file comparison using the software supply chain network. The keyword classifier tags diffs whose titles or tags contain terms like \"remove\", \"delete\", \"cleanup\", \"refactor\", \"dead\", or \"better engineering\", and yields the 14.2% share. The impact evaluation identifies reengineering diffs through task links, tracks all files they touch, and compares pre- and post-intervention periods for outage incidence, Diff Authoring Time (the time from first edit session to landing the diff), session count, cyclomatic complexity, and Katz centrality on a combined network of call graphs, co-changes, and author-to-file links. Control files are matched by size, language, and pre-intervention centrality so that adjusted centrality changes can be separated from global graph drift. The named identity at the center is Katz centrality, which scores a file by the importance of its neighbors rather than by degree alone.","core_discovery":"The central discovery is that code improvement at Meta is neither rare nor purely top-down: it is an ecology of organic diff-level cleanup, gamified Better Engineering engagement, and dedicated strategic initiatives, and it accounts for 14.2% of all diffs when identified through title and tag keywords. When a suborganization's six-month reengineering effort is evaluated by comparing files and diffs before and after, dead code removal and CCN-driven decompositions show 90% and 55% reductions in SEV-triggering diffs; median Diff Authoring Time drops by 41% to 77% across the four task types; editing sessions fall by 19% to 67%; cyclomatic complexity decreases by up to 26% for CCN-driven work; and, contrary to expectation, call-graph centrality increases. The authors interpret the centrality increase as an open question, possibly an artifact of co-change and author-to-file relationships that lag the intervention or of large-class splits creating more files. The paper claims this as a replication and extension of prior reengineering impact studies, with the new finding that a much larger fraction of changes is perfective than earlier estimates suggested.","pith_inferences":["If the 14.2% share is confirmed by manual inspection, it would imply that organic, developer-initiated improvement is a larger quality force than structured refactoring programs in large tech companies, since Better Engineering-tagged diffs account for only 4.3% of the total.","The centrality increase could be tested directly by recomputing adjusted centrality with a longer post-period, such as 12 months, to see whether co-change and authorship networks converge back down after the reengineering settles.","The prioritization criteria, especially normalized Diff Authoring Time and knowledge-loss share, could be packaged as a general tool for other companies: rank files by future-facing change load rather than by current smell counts, which is the paper's implicit design principle.","A cross-company replication using the same keyword classifier could settle whether the 14% versus 4% gap is a Meta-specific artifact or a sign that earlier estimates are outdated."],"forward_implications":["A company can spend over 14% of its change traffic on perfective maintenance without sacrificing release cadence.","Dead code removal and complexity-driven decompositions are the forms of reengineering most clearly associated with quality gains, with 90% and 55% reductions in the proportion of outage-triggering diffs.","Reengineering shortens authoring time for the touched files by 41% to 77%, which supports treating Diff Authoring Time as a primary return-on-investment metric for improvement work.","Prioritization should target actively changed, high-centrality, high-effort files rather than peripheral smelly code, because the expected benefit scales with future change frequency.","Call-graph centrality should not be assumed to decrease after reengineering; the paper's observed increases signal either measurement lag or a structural side effect of splitting large classes."],"supporting_citations":[{"why":"Geppert, Mockus, and Rösler's reengineering impact study is the replication target, supplying the defect-reduction and effort-reduction findings this paper extends.","marker":"[6]"},{"why":"Mockus and Votta's keyword-based diff classification approach is the method used to identify perfective maintenance diffs.","marker":"[26]"},{"why":"Kim, Zimmermann, and Nagappan's field study of refactoring provides the broad industry view of refactoring challenges and benefits that motivates the replication.","marker":"[8]"},{"why":"Software supply chain centrality is defined here, and it is used both as a prioritization metric and as an outcome measure in the impact evaluation.","marker":"[31]"},{"why":"Shackleton et al.'s dead code removal work at Meta documents the tooling foundation for the dead-code-removal task type.","marker":"[20]"},{"why":"Swanson's definition of perfective maintenance is the construct the keyword classifier operationalizes.","marker":"[25]"},{"why":"Katz centrality is the graph measure used for centrality comparisons before and after reengineering.","marker":"[34]"},{"why":"Potdar and Shihab's self-admitted technical debt keywords are used to check the mirror image of code improvement and to confirm its rarity.","marker":"[27]"}],"fun_headline_variants":["Meta's 14% cleanup rate cuts authoring time 77% and outages 90%","Code improvement at Meta: 14% of diffs, 90% fewer outages","Meta's 14% improvement diffs: 77% faster authoring, 90% fewer outages","How Meta keeps code healthy: 14% improvement work, fewer outages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline share of 14.2% rests on the assumption that diffs whose title or tags contain words like \"remove\", \"delete\", \"cleanup\", \"refactor\", \"dead\", or \"better engineering\" accurately capture all, and only, perfective maintenance.","fun_headline_variants_meta":{"raw":{"variants":["Meta's 14% cleanup rate cuts authoring time 77% and outages 90%","Code improvement at Meta: 14% of diffs, 90% fewer outages","Meta's 14% improvement diffs: 77% faster authoring, 90% fewer outages","How Meta keeps code healthy: 14% improvement work, fewer outages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000967,"raw_usage":{"total_tokens":4154,"prompt_tokens":1025,"completion_tokens":3129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":3035}},"tokens_in":641,"tokens_out":3129,"duration_ms":22602,"temperature":1.0,"reasoning_tokens":3035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:29:08.259077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of diffs at Meta that carry none of the improvement keywords and have engineers label them as perfective or not; if many are perfective despite lacking the keywords, the 14.2% share is a lower bound that could be much higher, while finding that most keyword-tagged diffs are actually feature work would deflate it. A sharper test would apply the same keyword methodology to a codebase where refactoring commits are recorded in a structured field and check whether the classifier recovers that known number.","supporting_citations":[{"cited_title":"Refactoring for changeability: A way to go?","cited_arxiv_id":null,"evidence_quote":"Geppert, Mockus, and Rösler's reengineering impact study is the replication target, supplying the defect-reduction and effort-reduction findings this paper extends."},{"cited_title":"Identifying reasons for software changes using historic databases,","cited_arxiv_id":null,"evidence_quote":"Mockus and Votta's keyword-based diff classification approach is the method used to identify perfective maintenance diffs."},{"cited_title":"A field study of refactoring challenges and benefits,","cited_arxiv_id":null,"evidence_quote":"Kim, Zimmermann, and Nagappan's field study of refactoring provides the broad industry view of refactoring challenges and benefits that motivates the replication."},{"cited_title":"Modeling the centrality of developer output with software supply chains,","cited_arxiv_id":null,"evidence_quote":"Software supply chain centrality is defined here, and it is used both as a prioritization metric and as an outcome measure in the impact evaluation."},{"cited_title":"Dead code removal at meta: Automatically deleting millions of lines of code and petabytes of deprecated data,","cited_arxiv_id":null,"evidence_quote":"Shackleton et al.'s dead code removal work at Meta documents the tooling foundation for the dead-code-removal task type."},{"cited_title":"The dimensions of maintenance,","cited_arxiv_id":null,"evidence_quote":"Swanson's definition of perfective maintenance is the construct the keyword classifier operationalizes."},{"cited_title":"An exploratory study on self-admitted technical debt,","cited_arxiv_id":null,"evidence_quote":"Potdar and Shihab's self-admitted technical debt keywords are used to check the mirror image of code improvement and to confirm its rarity."}],"review_version":1}