{"id":"8dcb6b21-8ef3-434a-91ce-d6afe0477171","arxiv_id":"2411.13777","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using 774,051 methods from 49 Java projects, this study finds that methods containing self-admitted technical debt comments are more complex, more change-prone, more bug-prone, and slower to have debt removed than methods without such comments.","lead":"This paper analyzes 774,051 Java methods from 49 open source projects and finds that methods with developer-admitted technical debt comments are larger, changed more often, and linked to more bug fixes. The result challenges earlier file-level studies that found no clear maintenance effect, suggesting granularity matters for seeing the cost of technical debt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Confound control is missing outside RQ2: the SATD versus NOT-SATD comparisons in RQ1 and RQ3 do not control for method age or size, so the claimed maintenance impact may be an artifact of older or larger methods being more likely to carry SATD comments.","rationale":"The reader's conditional verdict is well-founded. The central claim requires that the SATD group differ from the NOT-SATD group only in the presence of SATD. This is a causal counterfactual claim, but SATD is an observational property, not a randomized treatment. The paper's own RQ2 analysis normalizes for method age, showing that the authors are aware of age as a confound, yet RQ1 and RQ3 do not receive the same treatment. Given that RQ1 measures metrics at the most recent snapshot and RQ3 counts bug-fix associations over the method's lifetime, both are vulnerable to the fact that older methods have more time to become large, complex, and bug-fixed. The aggregated bug ratios are exactly the pattern age would produce. The project-level consistency does not remove the confound because the same age or size imbalance can occur project by project. I also considered whether the SATD detector's errors or CodeShovel's limitations could be a load-bearing flaw; these are acknowledged limitations, and random label noise would generally bias toward the null, making them less threatening to the observed positive findings. The strongest single testable issue remains confound control. The paper's shared dataset and method histories make the matched analysis feasible, so the condition can be resolved rather than merely debated.","tokens_in":20419,"tokens_out":3552,"duration_ms":85371,"concrete_test":"Use the shared dataset and CodeShovel histories to run a matched comparison for RQ1 and RQ3. For each SATD method, find a NOT-SATD method in the same project matched on introduction date (within, say, 90 days) and initial size (LOC within a tolerance or via nearest-neighbor matching), also controlling for initial complexity if feasible. Recompute the 14 RQ1 metrics and the three RQ3 bug ratios, including effect sizes, on the matched sample. As an alternative or complement, fit a logistic or mixed-effects model for bug-proneness with method age, size, and SATD as predictors. If the SATD effect collapses below 'small' in the matched sample, the central claim weakens; if it remains medium or large, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that methods that begin with SATD comments are more maintenance-prone than methods that never contain them. The most load-bearing unsecured assumption is that the SATD and NOT-SATD groups are comparable except for the presence of SATD. Section IV-B applies a 2-year age normalization only for change-proneness; RQ1 (Section IV-A) measures code metrics 'from the most recent versions' with no age or size control, and RQ3 (Section IV-C) compares bug ratios with no age, size, or other confound control. Because SATD comments are not randomly assigned, methods that start with SATD may simply be older, larger, or more complex to begin with, and older or larger methods have more opportunity to accrue revisions and bug-fix commits. The aggregated bug ratio gap (0.396 versus 0.213 in the HighRecall dataset) is exactly the pattern that age or size alone would produce. The same issue threatens RQ1: if SATD is more common in methods that were already large and complex at introduction, the cross-sectional metric differences in Table II are explained without invoking SATD as a cause. The project-level consistency in Tables III, V, and VII does not remove the confound, because the same age or size imbalance can recur project by project. Without matched or multivariate analysis, the causal interpretation in the abstract and discussion is not supported by the evidence presented. This does not mean the authors are wrong; it means the decisive evidence is not yet in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies 774,051 Java methods from 49 open-source projects, comparing methods that begin with self-admitted technical debt (SATD) comments to methods that never contain SATD comments. Four research questions assess code quality metrics (RQ1), change-proneness (RQ2), bug-proneness (RQ3), and SATD resolution time (RQ4). The authors report that SATD methods are larger, more complex, less readable, more change-prone, and more bug-prone, and that more than 61% of SATD comments are never removed. They conclude that SATD has a significant negative impact on method-level maintainability and should be proactively managed.","tokens_in":20746,"tokens_out":5125,"duration_ms":45602,"significance":"The contribution is potentially significant: it is one of the first large-scale method-level studies to link SATD to maintenance indicators, and it uses a substantially larger dataset than prior class-level studies, with method histories tracked by CodeShovel, a replication package on Zenodo, and multiple robustness checks (Benjamini-Yekutieli correction, three bug-labeling datasets, project-level analyses). If the observed associations are robust to confounding, the results would overturn prior null findings and support investment in SATD detection and remediation. However, the paper's central causal claim is not yet supported because the main comparisons do not control for method age or size, as detailed below.","major_comments":[{"comment":"The bug-proneness comparison in RQ3 compares raw bug ratios between SATD and NOT-SATD methods without controlling for method age or size. The paper itself notes in §IV-B that age is strongly correlated with change- and bug-proneness and applies a 2-year age normalization there, but no equivalent adjustment is made for the bug ratios in Table VI and Figure 3. The aggregated gap (0.396 vs 0.213 for HighRecall) is exactly what one would expect if SATD methods are older or larger on average, since older methods have longer exposure to bug-fix commits. The authors should use matching (e.g., on age, size, and introduction-time complexity) or a multivariate model with grouped project effects before claiming that SATD causes bug-proneness. As written, the abstract's claim of a 'higher tendency for bugs' is only an unadjusted association.","section":"§IV-C, RQ3"},{"comment":"For RQ1, code metrics are measured 'from the most recent versions of the methods' (Section IV-A.1), and no baseline at method introduction is reported. Because the labeling approach (Section III-D) classifies methods by whether they start with SATD, the observed cross-sectional differences in Table II could reflect pre-existing size, complexity, or age differences at the time of introduction rather than evolution caused by SATD. For example, a method may receive a SATD comment precisely because it is already large and complex. The authors should either report the same metric comparisons at the introduction version, or match methods on introduction-time size, complexity, and age. Without this, RQ1 does not support the summary statement that SATD 'degrad[es]' code quality over time.","section":"§IV-A, RQ1"},{"comment":"The threats-to-validity section does not mention the comparability of the SATD and NOT-SATD groups, which is the central threat to RQ1-RQ3. Internal validity is discussed only in terms of statistical tests and SATD comment identification, while the age/size confounding that the authors acknowledge and address in RQ2 is omitted for RQ1 and RQ3. The paper should include this as an explicit threat and provide the adjusted or matched results (or, if adjustment is infeasible with the current data, temper the causal wording throughout the abstract, RQ summaries, and conclusion).","section":"§V, Threats to Validity"}],"minor_comments":[{"comment":"The columns N, S, M, L appear to be computed only among projects with statistically significant differences, while P>0.05 is computed over all projects; please state this in the table caption or legend, since the row sums currently exceed 100% (e.g., Readability sums to 106.52).","section":"§IV-A, Table III"},{"comment":"The 2-year age normalization is described in one sentence but its effect is not quantified; report the results before and after normalization (or at least a summary in the text) so readers can evaluate the robustness claim.","section":"§IV-B"},{"comment":"Report the counts behind the bug ratios (number of bug-prone methods and total methods per dataset); the HighPrecision ratios are very small and counts would aid interpretation of the small effect size.","section":"§IV-C, Table VI"},{"comment":"The text and figures use both 'NOT-SATD' and 'NOT_SATD'; make the labeling consistent.","section":"Figures 1–3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and addresses a timely question. My main concern is the gap between the causal claim and the observational evidence: RQ1 and RQ3 lack the age/size control that the authors themselves apply in RQ2. I would recommend requesting a revision that adds confound control or substantially weakens the causal language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey [Name],\n\nRead the SATD paper. Here's the short version: it's the first method-level, large-scale look at whether self-admitted technical debt actually hurts maintenance, and it shows clear associations — SATD methods are bigger, more complex, less readable, more change-prone, more bug-prone, and their debts often persist for years. The dataset is substantial: 774k methods across 49 projects, and they've shared it. Using CodeShovel to track method histories is the right call and a real improvement over the file/class-level analyses that previously found no clear effect.\n\nWhat's genuinely new: the granularity. The prior work (Potdar & Shihab, Wehaibi et al., Bavota & Russo) worked at file or class level and largely came up empty. At method level, the signal is strong and consistent across metrics and projects. They also report effect sizes, not just p-values, and they check both SATD labeling approaches. RQ4's descriptive finding — over 60% of SATDs never get removed, and a fifth take more than 1000 days — is useful regardless of the causal question.\n\nNow the soft spots, and the stress-test note is on target. The causal claims in the abstract and discussion outrun the design. SATD comments aren't randomly assigned; methods that start with them might be older, larger, or more complex from day one. The paper only applies the 2-year age normalization in RQ2. RQ1 (code metrics) and RQ3 (bug ratio) compare raw groups with no matching or multivariate control. The bug ratio gap (0.396 vs 0.213) is exactly the pattern age or size alone could produce. Without a matched or regression analysis, the association is real but the causal interpretation isn't supported. Also, since they label methods by SATD in their initial version, selection effects are plausible: developers are more likely to write an admission comment on a method that's already messy. That's a real alternative explanation the paper doesn't test.\n\nMinor: the SATD detector is a text-mining tool with known accuracy limits, and the removal-time analysis treats comment disappearance as resolution, which they acknowledge in the threats. Both are okay as long as the language stays hedged; right now the abstract doesn't hedge.\n\nWho's this for? Anyone working on technical debt, code metrics, or MSR methodology. It deserves a serious referee — the dataset and method-level angle are worth engaging, and a revision with matched analyses and softened causal language would turn this into a solid reference. I'd bring it to reading group mostly to discuss how often this age/size confound shows up in mining studies. I wouldn't cite it for the causal claim in its current form, but I might cite the dataset or the RQ4 result.\n\nRecommendation: send it to review, with a clear request for the authors to add confound controls or at minimum rewrite the causal language.","headline":"Method-level SATD study with real associations, but missing age/size controls make the causal claims premature.","tokens_in":21239,"tokens_out":3417,"would_cite":false,"duration_ms":33149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-admitted technical debt comments mark methods that become larger, more complex, and more prone to bugs and changes.","keywords":["self-admitted technical debt","technical debt","method-level analysis","code metrics","bug-proneness","change-proneness","software maintenance","empirical software engineering"],"falsifier":"Take methods introduced in the same project and year with similar initial size and cyclomatic complexity, split them by whether the first version carries a SATD comment, and compare future revision counts and bug-fix rates; if the gap shrinks to nothing, the claim that SATD itself drives maintenance problems is falsified.","tokens_in":20253,"feed_emoji":"🐛","tokens_out":5692,"duration_ms":49606,"temperature":0.7,"pith_summary":"The paper asks whether self-admitted technical debt (SATD)—comments in which developers acknowledge substandard code—genuinely harms software maintenance, and it answers at the granularity of individual methods rather than whole files or classes. On 774,051 Java methods from 49 open-source projects, methods that begin life with a SATD comment end up significantly larger, more complex, less readable, and less maintainable than methods that never carry such comments, across all 14 code metrics studied. The same methods undergo more revisions and are more likely to be touched by bug-fixing commits, and the debt itself is rarely paid off: more than 60% of SATD comments are never removed, and 20% of those that are removed take over 1,000 days. If the claim holds, it supplies the missing empirical evidence that SATD is not just a label but a genuine maintenance liability, justifying early detection and remediation.","feed_headline":"Self-admitted debt comments predict buggier methods","feed_subtitle":"A 774,051-method study finds SATD methods are more change- and bug-prone, and most debts are never removed.","key_machinery":"The load-bearing mechanism is the method-level comparison design built on two tools. CodeShovel reconstructs each method's full history across renames, moves, and signature changes, so a method can be followed from birth to its current version; the SATD detector, a text-mining classifier, labels comments as SATD. A method counts as SATD if its initial version contains a flagged comment, and as NOT-SATD only if no version ever contains one, which lets the authors attribute later maintenance differences to the debt present at the method's origin. Code metrics, revision counts, edit distances, and bug-fix associations are then compared between the two groups with CDF plots, Wilcoxon rank-sum tests, and Cliff's delta effect sizes, with a 2-year age normalization applied to the change-proneness analysis.","core_discovery":"The central discovery is that SATD's widely assumed harm becomes visible once analysis moves from the file or class level to the method level. Methods whose first version contains a comment flagged as SATD by an automated detector are, at method level, statistically distinct from never-SATD methods: larger size, higher cyclomatic complexity, lower readability scores, worse maintainability index, more revisions and larger edit distances, and higher bug ratios under three different bug-definition datasets. The paper also finds the debt is persistent: with method history tracked across renames and moves, 61% of SATDs are never resolved, and the resolution-time distribution shows 60% of resolved debts taking at least 100 days and 20% taking more than 1,000 days. The authors present these results as the concrete evidence that earlier file- and class-level studies failed to find.","pith_inferences":["The same data could support a stronger test: match SATD and NOT-SATD methods on initial size and complexity at introduction, then compare their growth rates, which would isolate the debt's marginal effect from method size.","Because the paper pools all SATD types, a natural extension is to split by debt category (design, defect, documentation, requirement, test); effect sizes likely vary by type and would tell developers which debts to repay first.","The age normalization was applied only to change-proneness; repeating the readability and bug analyses on 2-year-normalized cohorts would show whether the quality gaps are age artifacts.","If developers write SATD comments under time pressure, the comment may be a proxy for rushed work; controlling for commit timing relative to releases would separate the debt signal from the haste signal."],"forward_implications":["A single comment in a method's first version is a cheap early warning that the method will consume disproportionate maintenance effort.","Because the differences show up in every one of 14 code metrics, SATD deserves a slot as a feature in bug- and change-prediction models, which the paper notes currently underuse it.","Long-lived debts, with 20% taking over 1,000 days to resolve, mean deferring SATD cleanup is likely to compound maintenance cost rather than save it.","The method-level granularity resolves the earlier null results from file- and class-level studies, so future SATD impact studies should operate at method level.","Practitioner-facing tools can flag methods with SATD comments as candidates for prioritized refactoring or testing."],"supporting_citations":[{"why":"Supplied the original SATD dataset and comment patterns; the file-level study that found no correlation between SATD and complexity metrics, which this paper challenges at method level.","marker":"[11]"},{"why":"Reported no clear defect–SATD trend at file level; the null result this method-level study revisits.","marker":"[25]"},{"why":"A large replication that found no code-quality correlation with SATD; another null result motivating the method-level lens.","marker":"[26]"},{"why":"CodeShovel, the method-history tracking tool that lets the authors follow methods across renames and moves and measure SATD removal times.","marker":"[30]"},{"why":"SATD detector, the text-mining tool that labels comments as SATD in this study.","marker":"[14]"},{"why":"Method-level bug prediction study whose labeling refinements (HighPrecision dataset) are reused for bug-proneness analysis.","marker":"[4]"},{"why":"Provided the 2-year age-normalization recommendation applied in the change-proneness analysis.","marker":"[9]"},{"why":"Earlier SATD removal study reporting median removal times of 18–172 days; the baseline for RQ4's more precise tracking.","marker":"[22]"}],"fun_headline_variants":["Method-level SATD predicts buggy code and never gets fixed","SATD at method level: more bugs, more changes, longer debt","Self-admitted technical debt: method-level bugs, long fixes","Method-level SATD: complex, buggy, and rarely paid off","Finally: method-level SATD shows real maintenance harm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes methods with and without SATD comments are otherwise alike, so if SATD methods tend to be older or larger from the start, the observed differences could come from age or size rather than from the debt itself.","fun_headline_variants_meta":{"raw":{"variants":["Method-level SATD predicts buggy code and never gets fixed","SATD at method level: more bugs, more changes, longer debt","Self-admitted technical debt: method-level bugs, long fixes","Method-level SATD: complex, buggy, and rarely paid off","Finally: method-level SATD shows real maintenance harm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3151,"prompt_tokens":934,"completion_tokens":2217,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":550,"tokens_out":2217,"duration_ms":15010,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:53:27.068702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take methods introduced in the same project and year with similar initial size and cyclomatic complexity, split them by whether the first version carries a SATD comment, and compare future revision counts and bug-fix rates; if the gap shrinks to nothing, the claim that SATD itself drives maintenance problems is falsified.","supporting_citations":[{"cited_title":"An exploratory study on self-admitted technical debt,","cited_arxiv_id":null,"evidence_quote":"Supplied the original SATD dataset and comment patterns; the file-level study that found no correlation between SATD and complexity metrics, which this paper challenges at method level."},{"cited_title":"Examining the impact of self- admitted technical debt on software quality,","cited_arxiv_id":null,"evidence_quote":"Reported no clear defect–SATD trend at file level; the null result this method-level study revisits."},{"cited_title":"A large-scale empirical study on self-admitted technical debt,","cited_arxiv_id":null,"evidence_quote":"A large replication that found no code-quality correlation with SATD; another null result motivating the method-level lens."},{"cited_title":"Codeshovel: Constructing method-level source code histories,","cited_arxiv_id":null,"evidence_quote":"CodeShovel, the method-history tracking tool that lets the authors follow methods across renames and moves and measure SATD removal times."},{"cited_title":"Satd detector: a text-mining-based self-admitted technical debt detection tool,","cited_arxiv_id":null,"evidence_quote":"SATD detector, the text-mining tool that labels comments as SATD in this study."},{"cited_title":"An empirical study on the removal of self-admitted technical debt,","cited_arxiv_id":null,"evidence_quote":"Earlier SATD removal study reporting median removal times of 18–172 days; the baseline for RQ4's more precise tracking."}],"review_version":1}