Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that automated SATD repayment is more feasible than prior work indicated, with prompt-based large language models correctly addressing 10.1% of Python and 8.1% of Java self-admitted technical debts exactly, roughly four…

desk verdict New large SATD repayment benchmarks and diff-based metrics are worth refereeing, but the headline EM numbers may be contaminated by unrelated edits in reference diffs. read the letter →

arxiv 2501.09888 v1 pith:NHF2HIKP submitted 2025-01-17 cs.SE

Mohammad Sadegh Sheikhaei , Yuan Tian , Shaowei Wang , Bowen Xu This is my paper · ORCID
classification cs.SE
keywords self-admittedtechnicaldebtSATDrepaymentlargelanguagemodelscodegenerationevaluationmetricsBLEU-diffLEMODbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to determine whether large language models can repay self-admitted technical debt (SATD) — the suboptimal code developers flag in comments like "TODO: optimize" — and how to measure that capability fairly. The authors build two large, filtered datasets of real SATD repayments (58,722 Python and 97,347 Java samples), argue that existing whole-code similarity metrics mislead, and propose evaluating only the diff between the method before and after repayment. Using BLEU-diff, CrystalBLEU-diff, and a new line-level metric (LEMOD), they find that prompt-based models like Gemma-2-9B address 10.1% of Python and 8.1% of Java SATDs exactly, far above the 2.3% previously reported. The work suggests automated SATD repayment is more achievable than earlier benchmarks implied, and that fine-tuned small models match large models on exact matches but produce worse code changes.

What carries the argument

The load-bearing mechanism is diff-based evaluation: instead of scoring similarity between the whole generated method and the whole ground-truth method, the paper computes the diff between the input method (containing the SATD) and each output, then scores those diffs. This yields BLEU-diff and CrystalBLEU-diff, and the new LEMOD metric, which computes precision, recall, and F1 over the set of changed lines in the reference diff versus the candidate diff. The other key object is the 10-step filtering pipeline, whose final step uses a zero-shot chain-of-thought prompt to Llama-3-70B to judge whether a method update actually repays the SATD, which removes roughly 60% of candidate samples as noise.

What would settle it

Manually inspect a random sample of the reference diffs (e.g., 300 items) and count the fraction containing edits unrelated to the SATD comment; if that fraction is high, BLEU-diff, CrystalBLEU-diff, and LEMOD would be rewarding or punishing models for reproducing irrelevant changes, and the reported rankings would need recomputation on purity-filtered diffs.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that automated SATD repayment is a tractable task for current LLMs once the data is clean and the evaluation looks at code changes rather than whole methods. The central discovery is twofold: first, a 10-step filtering pipeline built on the language-independent SATD Tracker and an LLM-as-judge relevance check removes noise that made prior datasets pessimistic, yielding datasets roughly twenty times larger than the only previous benchmark; second, evaluating on diffs changes the picture of model quality, since whole-code BLEU and CrystalBLEU correlate near zero with exact match while the diff-based metrics correlate 0.65–0.84. On these new benchmarks a simple prompt to Gemma-2-9B repays 10.1% of Python and 8.1% of Java SATDs exactly, versus 2.3% for the best earlier fine-tuned model, and larger prompt-based models lead on diff-based quality metrics while fine-tuned small models only match large ones on exact match and lose on the diff-based metrics.

Load-bearing premise

The whole evaluation assumes that the diff between the method before and after the commit contains exactly the changes that repay the SATD, with no unrelated edits; the LLM filter only checks whether the update is related to the SATD comment, not that every changed line is.

Editorial extensions

If this is right

  • If the central claim holds, automated SATD repayment at the method level is within reach of current prompt-based LLMs, with exact-match rates around 10% and much higher line-level overlap on easy cases.
  • Whole-code BLEU and CrystalBLEU should not be used to rank SATD repayment models, since they do not correlate with exact correctness; diff-based metrics or LEMOD should be the standard.
  • Fine-tuned small models (220M–770M parameters) can match large models on exact match but lag on change quality, so the choice of approach depends on whether exact replication or line-level fidelity matters more.
  • A cleaner dataset directly improves both prompt-based and fine-tuned performance, as shown by the RQ3 comparison on the filtered version of the prior Java-only dataset.
  • An oracle that picks the best prompt per SATD item raises exact match to 11–14%, suggesting prompt selection is a major lever for further gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The diff-purity assumption could be tested directly: manually check a random sample of reference diffs for edits unrelated to the SATD, and re-evaluate models on a subset restricted to pure diffs to see how much the reported scores change.
  • The LEMOD metric, being interpretable, could transfer to other code-editing tasks, such as program repair or refactoring, where the edit diff matters more than the whole file.
  • The paper's easy-versus-hard split suggests a practical triage: models reliably handle SATDs requiring one or two inserted lines, so tools could route only those to automation and leave larger refactors to humans.
  • The oracle-template result implies that learning a prompt router — a small classifier that picks the prompt per SATD — could push exact match well beyond any single prompt, since 15–20% of items are addressed by at least one model-prompt combination in the 1,000-sample study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces two large-scale SATD-repayment datasets (58,722 Python and 97,347 Java items), constructed with an SATD tracking tool and a 10-step filtering pipeline whose final step uses an LLM judge to remove updates unrelated to SATD repayment. It proposes three diff-based evaluation metrics—BLEU-diff, CrystalBLEU-diff, and LEMOD—and uses them, together with EM, to compare fine-tuned CodeT5p models against five prompt-based LLMs across four prompt templates. The main empirical findings are that fine-tuned small models achieve EM scores comparable to prompt-based approaches while lagging on the fine-grained diff metrics, and that Gemma-2-9B with a simple prompt achieves the best EM (10.1% Python, 8.1% Java), well above the 2.3% reported by prior work.

Significance. If the central claims hold, this is a substantial empirical contribution to automated SATD repayment. The paper contributes large public datasets, a reproducible evaluation pipeline, and new metrics that correlate strongly with EM, and it provides the first multi-language comparison of fine-tuned and prompt-based LLMs on this task. The finding that fine-tuned small models match prompt-based models on EM but underperform on finer-grained diff metrics is useful and non-obvious, and the large improvement over the previous 2.3% EM baseline would change the practical outlook for the task. The availability of datasets and source code is a clear strength, as is the attention to metric interpretability.

major comments (3)
  1. [Section 3.1 (filter step 10) and Section 3.2 (reference_diff)] The benchmark ground truth is the full method diff, not the lines that actually repay the SATD. The step-10 filter asks Llama-3-70B whether the whole method update is related to the SATD comment; Table 5 reports precision of 0.830 (Python) and 0.926 (Java) on the predicted 'Yes' items, so 8-17% of accepted items can still be false positives, and even true repayments often co-occur with unrelated edits, as the paper notes from Zampetti et al. [39]. Because EM, BLEU-diff, CrystalBLEU-diff, and LEMOD are all computed against reference_diff = diff(input_code, ground_truth), any unrelated changed line must be reproduced exactly for EM=1 and contributes to every diff-based score. Section 7 acknowledges that 'some items with incorrect ground truth remain' but does not quantify line-level contamination or show that the headline results in Tables 12/13 survive it. I ask for a line-level purity analysis on a random sample, and a recomputation of the RQ4 results using reference diffs restricted to SATD-related lines.
  2. [Section 3.2 (RQ2) and Section 3.4 (RQ4)] The 1,000 RQ2 samples are selected 'from each dataset' without any explicit statement about their relation to the repository-wise split used in RQ4. The per-model best prompt identified in RQ2 (Tables 7/8) is then used to produce the RQ4 rankings in Tables 12/13. If the RQ2 sample overlaps the RQ4 test set, the prompt selection is informed by test data, which can inflate RQ4 scores and potentially change the comparison between fine-tuned and prompt-based models. The manuscript must state explicitly whether the RQ2 samples are a subset of the RQ4 test set or are drawn disjointly from it, and should preferably select prompts on training or validation data only.
  3. [Section 3.2 (BLEU-diff and CrystalBLEU-diff)] The operational definitions of BLEU-diff and CrystalBLEU-diff are underspecified. The text says that diffs are computed with difflib and that BLEU/CrystalBLEU are then calculated 'on these diffs,' but it does not specify whether deleted lines, inserted lines, or a concatenation of both are used, how the brevity penalty applies to a diff string, or how line-level matches are tokenized. Since these metrics are central to the RQ4 comparison and are claimed as a contribution, the definitions need to be precise enough to reproduce Tables 12/13 from the paper alone.
minor comments (6)
  1. [Table 3 caption] The caption refers to the 'Maldonado dataset,' but the comparison is with the Mastropaolo dataset; please correct the attribution.
  2. [Section 7 (Internal Validity)] The statement that the precision and recall errors of the filter 'counteract each other' is not justified; precision and recall errors can bias scores in either direction and should be quantified separately rather than assumed to cancel.
  3. [Section 5.2, Table 6] The baseline row uses the input code after ICD removal, which is not a model output; please state explicitly that this baseline is illustrative and is not part of a statistical comparison among models.
  4. [Section 3.1] The paper alternates between 'adopt' and 'design' for the 10-step filtering pipeline; since steps 1-8 are explicitly taken from Mastropaolo et al., the contribution should be phrased as the two new filtering steps (9 and 10).
  5. [Table 18] The layout of the correlation matrix is confusing because the Python and Java results are combined in one table with unclear upper/lower triangle labels; please present two separate matrices or label the triangles unambiguously.
  6. [Section 6.1] The oracle-template results are an upper bound and should be accompanied by an explicit caveat that they are not attainable without a working selector.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-level circularity: the reported predictions are scored against developer ground-truth diffs, and the only overlapping-authors citation (SATD Tracker) is a data-collection tool whose performance is re-reported in this paper rather than a result that defines the outcome.

full rationale

The paper's central claims (RQ2-RQ4) are evaluated by comparing LLM-generated code against developer ground-truth diffs. EM, BLEU-diff, CrystalBLEU-diff, and LEMOD are all defined in Section 3.2 from reference_diff = diff(input_code, ground_truth) and candidate diffs of model outputs; none of these metrics contains a fitted parameter that is later renamed as a prediction. The best-prompt selection in RQ2 is performed on a 1,000-item sampled set and then applied to held-out test splits (Table 2), and RQ3 uses the external Mastropaolo dataset, so no target result is used to construct the inputs. The only self-citation that is potentially load-bearing is SATD Tracker [33], used to mine the benchmark; however, the paper independently reports its processing success rates (97.5% Python, 96.4% Java in Table 4) and uses it as a mining tool rather than as an argument that forces the reported EM values. The Llama-3-70B relevance filter (Section 3.1, step 10) is a data-cleaning step, not a fitted parameter for downstream repayment predictions; it may bias model-family comparisons because Llama-3.1 models are also evaluated, but this is a validity concern rather than a circular derivation. Section 7 explicitly acknowledges the filter's imperfect precision ('some items with incorrect ground truth remain') and notes the counteracting recall threat, which is an honest limitation, not a self-referential proof. The oracle-template analysis in Section 6.1 is presented as a hypothetical upper bound, not as a claimed predictive result. Therefore, no step in the derivation chain reduces by construction to its own inputs, and the appropriate score is 1: a mild self-reference without load-bearing circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on dataset cleanliness, the validity of the ground-truth diff, and the use of EM as a reference metric. The only hand-chosen numeric threshold in the analysis is the easy/hard SATD split. No parameters are fitted to the target result.

free parameters (1)
  • easy/hard SATD split threshold = at most 2 inserted lines (easy); at least 3 inserted lines (hard)
    The authors partition the test set by the number of inserted lines in the ground truth diff. This chosen threshold drives the analysis that EM is unreliable on hard SATDs and that fine-tuned models fall behind on hard items.
assumptions (4)
  • domain assumption The ground truth diff (developer's repaid method vs. input) contains only changes related to the SATD repayment.
    Used in Section 3.2 to define reference_diff and in all metric calculations; if false, metrics are biased.
  • domain assumption The Llama-3-70B LLM judge, validated on 200 manually labeled samples, generalizes across the full dataset and correctly identifies SATD-relevant updates.
    Step 10 of the filtering pipeline in Section 3.1 relies on this filter for dataset cleanliness.
  • domain assumption Exact Match (EM) is an intuitive, reliable gold standard against which other metrics should be validated.
    Section 3.2 uses correlation with EM as the measure of metric effectiveness; if EM is itself flawed, the validation is weak.
  • domain assumption The repository selection criteria and filtering steps produce a representative sample of SATD repayments, excluding class-level and file-level SATDs.
    Section 3.1 restricts to method-level SATDs and excludes long methods and non-ASCII comments, which limits generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment." pith.science (2026). https://pith.science/paper/NHF2HIKP

@misc{pith2026250109888,
  author       = {Pith},
  title        = {Pith review of: Understanding the Effectiveness of LLMs in Automated Self-Admitted Technical Debt Repayment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHF2HIKP}},
  note         = {Machine review of arXiv:2501.09888}
}
read the original abstract

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality and increase maintenance costs. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayments, and (3) limitations of existing evaluation metrics. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repositories, resulting two large-scale datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code. Additionally, we propose another new metric, LEMOD, which is both interpretable and informative. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models, and prompt engineering with five large-scale models. Our results reveal that fine-tuned small models achieve comparable Exact Match (EM) scores to prompt-based approaches but underperform on BLEU-based metrics and LEMOD. Notably, Gemma-2-9B leads in EM, addressing 10.1% of Python and 8.1% of Java SATDs, while Llama-3.1-70B-Instruct and GPT-4o-mini excel on BLEU-diff, CrystalBLEU-diff, and LEMOD metrics. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.

Figures

Figures reproduced from arXiv: 2501.09888 by the authors.

Figure 1
Figure 1. Example of a Java SATD removal where code update is unrelated to SATD repayment [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompt used in final filtering step to detect relevance of code updates to SATD repayment [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of BLEU and CrystalBLEU scores on the whole code versus on the diffs. [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 16 canonical work pages

  1. [39]

    Fiorella Zampetti, Alexander Serebrenik, and Massimiliano Di Penta. 2018. Was self-admitted technical debt removal a real removal? an in-depth perspective. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New York, NY, USA, 526–536. https://doi.org/10.1...

    arXiv 2018

  2. [1]

    Eman Abdullah AlOmar, Ben Christians, Mihal Busho, Ahmed Hamad AlKhalid, Ali Ouni, Christian Newman, and Mohamed Wiem Mkaouer. 2022. SATDBailiff-mining and tracking self-admitted technical debt. Science of Computer Programming 213 (2022), 102693. https://doi.org/10.1016/j.scico. 2021.102693

    arXiv 2022

  3. [2]

    Aaditya Bhatia, Foutse Khomh, Bram Adams, and Ahmed E Hassan. 2023. An Empirical Study of Self-Admitted Technical Debt in Machine Learning Software. arXiv:2311.12019 [cs.SE]

    arXiv 2023

  4. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  5. [4]

    Frank Buschmann. 2011. To Pay or Not to Pay Technical Debt. IEEE Software 28, 6 (2011), 29–31. https://doi.org/10.1109/MS.2011.150

    doi:10.1109/ms.2011.150 2011

  6. [5]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation . IEEE Transactions on Software Engineering 49, 07 (J...

    doi:10.1109/tse.2023 2023

  7. [6]

    LMarena AI

    Chatbot Arena LLM Leaderboard 2024. LMarena AI. https://lmarena.ai/ Accessed: 2024-07-31

    work page 2024

  8. [7]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2024. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv:2310.14735 [cs.CL] https://arxiv.org/abs/2310.14735

    arXiv 2024

Show all 40 references
  1. [8]

    Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder...

    2021 arXiv

  2. [9]

    Shaiful Chowdhury, Hisham Kidwai, and Muhammad Asaduzzaman. 2024. Evidence is All We Need: Do Self-Admitted Technical Debts Impact Method-Level Maintenance? arXiv:2411.13777 [cs.SE] https://arxiv.org/abs/2411.13777

    2024 arXiv

  3. [10]

    Ward Cunningham. 1992. The WyCash Portfolio Management System. In Addendum to the Proceedings on Object-Oriented Programming Systems, Languages, and Applications (Addendum) (Vancouver, British Columbia, Canada) (OOPSLA ’92). Association for Computing Machinery, New York, NY, U...

    1992

  4. [11]

    Ozren Dabic, Emad Aghajani, and Gabriele Bavota. 2021. Sampling Projects in GitHub for MSR Studies. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . 560–564. https://doi.org/10.1109/MSR52588.2021.00074 Manuscript submitted to ACM Understan...

    2021

  5. [12]

    Aryaz Eghbali and Michael Pradel. 2023. CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE ’22). Association for Computing Machinery, ...

    2023

  6. [13]

    Zhaoqiang Guo, Shiran Liu, Jinping Liu, Yanhui Li, Lin Chen, Hongmin Lu, and Yuming Zhou. 2021. How Far Have We Progressed in Identifying Self-Admitted Technical Debts? A Comprehensive Empirical Study. ACM Trans. Softw. Eng. Methodol. 30, 4, Article 45 (jul 2021), 56 pages. ht...

    2021 doi

  7. [14]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. (Sept. 2024). https://doi.org/10.1145/36959...

    2024 doi

  8. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  9. [16]

    Yikun Li, Mohamed Soliman, and Paris Avgeriou. 2023. Automatically Estimating the Effort Required to Repay Self-Admitted Technical Debt. arXiv:2309.06020 [cs.SE]

    2023 arXiv

  10. [17]

    Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, and Shanping Li. 2021. An Exploratory Study on the Introduction and Removal of Different Types of Technical Debt. Empirical Software Engineering 26 (2021), 16

    2021

  11. [18]

    Jiaxing Liu, Chaofeng Sha, and Xin Peng. 2023. An Empirical Study of Parameter-Efficient Fine-Tuning Methods for Pre-Trained Code Models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 397–408

    2023

  12. [19]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126 (2024)

    2024 arXiv

  13. [20]

    Marcos Macedo, Yuan Tian, Filipe Cogo, and Bram Adams. 2024. Exploring the Impact of the Output Format on the Evaluation of Large Language Models for Code Translation. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engin...

    2024

  14. [21]

    Everton Da S Maldonado, Rabe Abdalkareem, Emad Shihab, and Alexander Serebrenik. 2017. An empirical study on the removal of self-admitted technical debt. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 238–248

    2017

  15. [22]

    Everton da Silva Maldonado and Emad Shihab. 2015. Detecting and quantifying different types of self-admitted technical Debt. In 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD) . 9–15. https://doi.org/10.1109/MTD.2015.7332619

    2015

  16. [23]

    Antonio Mastropaolo, Massimiliano Di Penta, and Gabriele Bavota. 2023. Towards Automatically Addressing Self-Admitted Technical Debt: How Far Are We? . In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, Los Alamitos,...

    2023

  17. [24]

    David OBrien, Sumon Biswas, Sayem Imtiaz, Rabe Abdalkareem, Emad Shihab, and Hridesh Rajan. 2022. 23 Shades of Self-Admitted Technical Debt: An Empirical Study on Machine Learning Software. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Sympo...

    2022

  18. [25]

    David OBrien, Sumon Biswas, Sayem Mohammad Imtiaz, Rabe Abdalkareem, Emad Shihab, and Hridesh Rajan. 2024. Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot. In Proceedings of the IEEE/ACM 46th International Conference on Software Engine...

    2024

  19. [26]

    Diogo Pina, Carolyn Seaman, and Alfredo Goldman. 2022. Technical debt prioritization: a developer’s perspective. In Proceedings of the International Conference on Technical Debt. 46–55

    2022

  20. [27]

    Andrea Pinna, Maria Ilaria Lunesu, Stefano Orrù, and Roberto Tonelli. 2023. Investigation on Self-Admitted Technical Debt in Open-Source Blockchain Projects. Future Internet 15, 7 (2023). https://doi.org/10.3390/fi15070232

    2023 doi

  21. [28]

    Aniket Potdar and Emad Shihab. 2014. An Exploratory Study on Self-Admitted Technical Debt. In 2014 IEEE International Conference on Software Maintenance and Evolution. 91–100. https://doi.org/10.1109/ICSME.2014.31

    2014 doi

  22. [29]

    Prenner and R

    J. Prenner and R. Robbes. 2022. Making the Most of Small Software Engineering Datasets With Modern Machine Learning. IEEE Transactions on Software Engineering 48, 12 (dec 2022), 5050–5067. https://doi.org/10.1109/TSE.2021.3135465

    2022

  23. [30]

    Python Software Foundation. 2024. difflib - Helpers for computing deltas . https://docs.python.org/3/library/difflib.html Python standard library

    2024

  24. [31]

    Xiaoxue Ren, Zhenchang Xing, Xin Xia, David Lo, Xinyu Wang, and John Grundy. 2019. Neural Network-Based Detection of Self-Admitted Technical Debt: From Performance to Explainability. 28, 3, Article 15 (jul 2019), 45 pages. https://doi.org/10.1145/3324916

    2019 doi

  25. [32]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Sohel Mondal, and Aman Chadha. 2024. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. ArXiv abs/2402.07927 (2024). https://api.semanticscholar.org/CorpusID: 267636769

    2024 arXiv

  26. [33]

    Mohammad Sadegh Sheikhaei and Yuan Tian. 2023. Automated Self-Admitted Technical Debt Tracking at Commit-Level: A Language-independent Approach. In 2023 ACM/IEEE International Conference on Technical Debt (TechDebt) . 22–26. https://doi.org/10.1109/TechDebt59074.2023.00009

    2023

  27. [34]

    Mohammad Sadegh Sheikhaei, Yuan Tian, SHAOWEI WANG, and Bowen Xu. 2024. An empirical study on the effectiveness of large language models for SATD identification and classification. Empirical Software Engineering 29, 6 (oct 2024). https://doi.org/10.1007/s10664-024-10548-3 Manu...

    2024 doi

  28. [35]

    Murali Sridharan, Leevi Rantala, and Mika Mäntylä. 2023. PENTACET data-23 Million Contextual Code Comments and 250,000 SATD comments. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 412–416

    2023

  29. [36]

    Jie Tan, Daniel Feitosa, and Paris Avgeriou. 2021. Do practitioners intentionally self-fix technical debt and why?. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 251–262

    2021

  30. [37]

    Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino,...

    2023 doi

  31. [38]

    Jerin Yasmin, Mohammad Sadegh Sheikhaei, and Yuan Tian. 2022. A first look at duplicate and near-duplicate self-admitted technical debt comments. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension . 614–618

    2022

  32. [40]

    Fiorella Zampetti, Alexander Serebrenik, and Massimiliano Di Penta. 2020. Automatically Learning Patterns for Self-Admitted Technical Debt Removal. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . 355–366. https://doi.org/1...

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.