Pith. sign in

REVIEW 4 major objections 5 minor 88 references

Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dependency-aware prompt that feeds helper-method bodies to an LLM produces measurably better comments for the majority of Java methods that call other methods.

desk verdict A useful dependency-labeled dataset and a believable RQ1/RQ2 story, but the causal claim that helper-method context improves comments is not yet supported by the numbers. read the letter →

arxiv 2504.19459 v1 pith:TH3P7M5O submitted 2025-04-28 cs.SE

Md Mustakim Billah , Md Shamimur Rahman , Banani Roy This is my paper · ORCID
classification cs.SE
keywords methoddependenciescodecommentgenerationsummarizationhelpermethodslargelanguagemodelsin-contextlearningunderstandingJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that method dependencies—one method calling another—are a hidden reason why automatic comment generators produce weak or generic comments. In ten large, active Java projects, dependent methods make up 69.25% of all methods, are changed more often, and draw more developer engagement than independent methods. Existing fine-tuned and LLM-based comment generators score noticeably lower on dependent methods, and the paper's proposed technique, HelpCOM, closes much of that gap simply by adding the bodies of the called helper methods to the prompt. If the claim holds, dependency-aware prompt construction is a cheap, broadly applicable fix for a problem that affects the majority of methods in real code.

What carries the argument

The load-bearing mechanism is prompt construction: HelpCOM retrieves the helper methods called by a dependent method, writes the dependent method's body followed by the helper bodies at each level, and instructs the LLM to output only the summary comment. Dependency classification is built by parsing Java files and matching method-invocation call sites to stored method definitions by method name and parameter count; HelpCOM1 includes only immediate helpers, while HelpCOMN includes the full transitive helper chain. This turns method dependency into extra in-context evidence, letting the LLM reason about what the helpers contribute before writing the comment.

What would settle it

Re-run the HelpCOM comparison on a sample where helper-method relationships are verified by a build tool or dynamic call tracing, and include a control prompt with unrelated method bodies; if the advantage over the same LLM without helpers drops below the reported 5.6% or disappears, the dependency-specific claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that method dependencies are the hidden variable in automatic comment generation. In ten large Java projects, 69.25% of methods call at least one other method, and these dependent methods account for 77.37% of commits and show higher author engagement than independent methods. Across fine-tuned code models and LLM-based summarizers, comments for dependent methods score 7.3% to 13.5% lower on the paper's overall metric than comments for independent methods. The proposed technique, HelpCOM, places the bodies of helper methods into the prompt alongside the dependent method and asks the LLM to write a Javadoc-style summary; the full-chain version outperformed every baseline by 5.6% to 50.4% on the overall metric combining syntactic, semantic, and LLM-based scores, and about 75% of 156 practitioners chose HelpCOM's comment in a blinded comparison.

Load-bearing premise

The load-bearing premise is that matching called method names plus parameter counts to method definitions correctly identifies which methods are dependent and which bodies are the relevant helpers; if that graph is noisy, both the baseline gap and HelpCOM's measured advantage could be artifacts of the matching rather than of genuine method dependencies.

Editorial extensions

If this is right

  • Dependent methods are the common case, not the exception: roughly 69% of methods in the studied projects call other methods, so single-method comment generators are working on the majority case without the context it needs.
  • Existing fine-tuned and LLM-based summarizers score 7.3% to 13.5% higher on independent methods than on dependent ones, meaning benchmark results over mixed sets overstate what developers will experience on real code.
  • Adding helper-method bodies to the prompt improved the best overall metric by 5.6% to 50.4% over baselines, and the improvement held whether immediate helpers or the full helper chain were included.
  • The practitioner survey supports the same conclusion: about 75% preferred HelpCOM's comment, and participants rated dependent methods easier to understand once helper definitions were visible.
  • The same prompting approach transferred to Python and PHP samples with higher overall scores than on the Java sample, suggesting the mechanism is not Java-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because helper matching relies on name and parameter count, the 69% figure and the reported performance gap are upper-bound estimates of dependency effects; a precise call graph from build artifacts or dynamic traces could change both.
  • A natural control experiment is to append unrelated method bodies to the same prompt; if comment quality rises equally, the gain is generic extra context rather than dependency information specifically.
  • The low end of the 5.6%–50.4% range may reflect methods with shallow dependencies, while the high end may apply where helper bodies carry most of the meaning; tool builders should expect context relevance to matter most for delegating methods.
  • An iterative extension would generate a first comment, let the LLM request specific helper bodies, then revise; the paper's static chain inclusion is one point on that design space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether method dependencies influence automatic comment generation. It builds a dataset of 647,769 methods from 10 popular Java GitHub projects, classifies methods as dependent or independent based on a Tree-sitter-based call-graph construction, and reports three results: (RQ1) dependent methods are more frequent (69.25%) and more change-prone; (RQ2) existing baselines (CodeT5+, CodeBERT, ASAP) produce worse comments for dependent than for independent methods; and (RQ3) a proposed prompt-augmentation technique, HelpCOM, which adds helper-method bodies to the prompt, improves comment quality for dependent methods. The headline claim is that HelpCOM outperforms baselines by 5.6% to 50.4% in an author-defined Overall Metric Score, and a survey of 156 practitioners reports that approximately 75% prefer HelpCOM-generated comments.

Significance. If the central causal claim is established, the paper identifies a practically important blind spot in code summarization (the prevalence and difficulty of dependent methods) and shows that a cheap, model-agnostic prompt change materially improves generated comments. The paper has clear strengths: it releases a replication package, constructs a large real-world dataset (647,769 methods from 10 active projects), uses git history for RQ1, employs multiple automatic metrics plus a 156-participant practitioner survey, and includes pilot cross-language experiments for Python and PHP. The main weakness is that the causal mechanism (helper-method context causes the improvement) is not yet supported by rigorous statistics against the proper control: the same LLM without helper context. The OMS weights and the SIDE-based filtering/ground-truth revision also need robustness analysis to make the quantitative claims credible.

major comments (4)
  1. [3.3, Table 4] The central claim that helper-method context is responsible for the improvement requires a comparison of HelpCOMN (GPT-4o) with plain GPT-4o, which differ only by the presence of helper-method bodies in the prompt. In Table 4, that contrast is small: OMSssl is 56.82 vs. 55.70 and OMSss is 40.35 vs. 39.33. The asterisks in Table 4 mark only METEOR and CIDEr as significant differences for this pair; BLEU, ROUGE-L, SBERT, USEnc, SIDE, and both LLM-based judges show no significant difference, and no significance test or confidence interval is reported for the OMS scores themselves. The abstract's 5.6%-50.4% range is computed against CodeT5+, CodeBERT, and ASAP, not against the same-LLM control, so it does not support the mechanism claim. The paper should report significance tests and effect sizes for the GPT-4o vs. HelpCOMN contrast, including for OMS, and should clearly distinguish the mechanism result from the overall superiority of a strong LLM baseline.
  2. [3.2-3.3] The RQ3 evaluation is performed on a dataset filtered by SIDE >= 0.8 (a model-based quality threshold) and on ground-truth comments that were revised by a senior developer after the original comments were rated as 'unacceptable'. This means the quantitative comparisons in Table 4 are not on the original, unmodified dataset. The authors should justify the choice of the 0.8 threshold, report how many methods were excluded, and provide a sensitivity analysis (e.g., without the SIDE filter, with different thresholds, and before vs. after ground-truth revision) to establish that the observed improvements in Table 4 are not an artifact of this selection or revision procedure.
  3. [2.4.1] The dependency graph that defines dependent and independent methods is built by matching Tree-sitter method_invocation nodes to previously extracted methods using only the method name and parameter count. This matching cannot resolve overloading, inheritance, dynamic dispatch, or calls to methods defined outside the repository, so helper methods can be silently mis-assigned or missed. Since RQ2 and RQ3 both compare dependent vs. independent methods, this categorization is load-bearing. The authors should validate the call-graph construction on a random sample (e.g., manual inspection of, say, 100-200 methods) and quantify the misclassification rate, or at least discuss how prevalent such cases are in the selected projects.
  4. [2.7.4] The Overall Metric Score (Eqs. 1-2) uses author-defined weights (0.46/0.54 for OMSss and 0.30/0.35/0.35 for OMSssl) with no sensitivity analysis. Since the central headline range '5.6% to 50.4%' is an OMS improvement, the ranking of HelpCOMN (GPT-4o) over plain GPT-4o and over the baselines may depend on these arbitrary weights. The authors should report how the OMS ranking changes under reasonable weight variations and, more importantly, show which individual metrics drive the improvement, so that the reader can judge the robustness of the headline claim.
minor comments (5)
  1. [7] Section 7 says 'we proposed a noble technique'; this should be 'novel technique'.
  2. [3.3] The inter-rater agreement is reported as 'Fleiss' Kappa score of 83.6% (above the 70% threshold)'. Fleiss' kappa is a coefficient (usually between 0 and 1) rather than a percentage; please clarify whether the reported value is percent agreement or kappa multiplied by 100, and describe how the 70% threshold was determined.
  3. [4.3] The practitioner survey compares HelpCOM only with CodeT5+ and ASAP, not with plain GPT-4o. As a result, the 75% preference for HelpCOM does not isolate the effect of helper-method context; including plain GPT-4o as an anonymous option in the survey would strengthen the mechanistic interpretation.
  4. [2.4.1] The dataset description reports 69,058 Java files and 647,769 methods, but there is no per-project breakdown. Reporting the number of files, methods, dependent methods, and helpers per repository would improve transparency and reproducibility.
  5. [2.3, Table 1] The repository versions are not pinned: the paper says 'main' or 'master' branches were cloned, but does not report commit hashes or retrieval dates. This makes exact replication difficult.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HelpCOM's helper-context effect is an independent empirical comparison, not a fitted or self-referential derivation.

full rationale

The paper's central claim is empirical: adding helper-method bodies to an LLM prompt improves generated comments for dependent methods. This is tested by a controlled comparison in Table 4 between plain GPT-4o and HelpCOMN (GPT-4o), which differ only in the injected helper context, and no parameter of HelpCOM is fitted to the evaluation metrics. The Overall Metric Score equations (1) and (2) are author-defined weighted averages, but they are not calibrated to make HelpCOM win; the reported 5.6% to 50.4% improvement is arithmetic over observed scores from three classes of metrics. The classification of methods as dependent or independent rests on Tree-sitter method-invocation extraction with name and parameter-count matching (Section 2.4.1), which may be noisy, but that is a validity threat rather than a circular reduction of the conclusion into its premise. The use of GPT-4o for generation and GPT-4/Llama-3.3 as LLM judges, and the use of SIDE for dataset filtering and later evaluation, are self-referential measurement choices that could bias absolute scores; however, they are applied uniformly to all compared systems, so the relative comparison does not reduce by construction to its inputs. No fitted input is renamed as a prediction, no load-bearing claim is justified solely by a self-citation, and no uniqueness or ansatz is imported from the authors' prior work. The weaknesses noted by a skeptical reader, such as the small and largely non-significant gap between plain GPT-4o and HelpCOMN on several metrics and the survey's omission of plain GPT-4o as a comparator, are correctness and evidence-strength concerns, not circularity. Accordingly, the derivation chain is self-contained and the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The study is empirical, so it introduces no mathematical axioms. The central claims rest on several domain assumptions: the parser correctly extracts methods and calls, the name-and-arity matching reconstructs the call graph, the LLM-as-judge prompt is a valid proxy for human quality, and the hand-revised ground-truth comments are a fair reference. The free parameters are evaluation choices (SIDE threshold, OMS weights, temperature, sample size) rather than fitted model parameters. No invented entities are introduced.

free parameters (4)
  • OMS weight coefficients = 0.46, 0.54 (Eq. 1); 0.30, 0.35, 0.35 (Eq. 2)
    Hand-chosen weights for the overall metric score; not derived from data, and the paper's description of the ratio (30:35) does not match Eq. 1.
  • SIDE threshold = 0.8
    Used to filter the evaluation dataset; a post-hoc choice that determines which methods are scored in RQ3.
  • LLM temperature = 0.2
    Set to reduce randomness in GPT-4o generations; affects reproducibility of all generated comments.
  • Sampling margin = 95% confidence, 5% margin of error, 380 methods per class
    Used to down-sample the dataset for LLM-based evaluation; the sample size is a researcher choice.
assumptions (4)
  • domain assumption Tree-sitter method_declaration and method_invocation nodes correctly identify all methods and calls in Java files.
    Method extraction and dependency classification in Section 2.4.1 depend entirely on the parser's accuracy.
  • domain assumption Matching calls to helper method bodies by method name and parameter count is unambiguous.
    Overloaded methods, inherited methods, and same-name static imports could cause wrong helper bodies to be attached, affecting HelpCOM's prompts (Section 2.4.1).
  • domain assumption The LLM-as-judge prompt from Sun et al. provides a valid proxy for human comment quality.
    LLM evaluation scores (GPT-4, Llama-3.3) are used as a full metric category in the OMS (Section 2.7.3).
  • domain assumption The senior developer's revision of ground-truth comments produces a better reference than the original developer comments.
    RQ3 metrics are computed against the revised comments (Section 3.3), so the ranking of systems is relative to this revised reference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding." pith.science (2026). https://pith.science/paper/TH3P7M5O

@misc{pith2026250419459,
  author       = {Pith},
  title        = {Pith review of: Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TH3P7M5O}},
  note         = {Machine review of arXiv:2504.19459}
}
read the original abstract

Method-level comments are critical for improving code comprehension and supporting software maintenance. With advancements in large language models (LLMs), automated comment generation has become a major research focus. However, existing approaches often overlook method dependencies, where one method relies on or calls others, affecting comment quality and code understandability. This study investigates the prevalence and impact of dependent methods in software projects and introduces a dependency-aware approach for method-level comment generation. Analyzing a dataset of 10 popular Java GitHub projects, we found that dependent methods account for 69.25% of all methods and exhibit higher engagement and change proneness compared to independent methods. Across 448K dependent and 199K independent methods, we observed that state-of-the-art fine-tuned models (e.g., CodeT5+, CodeBERT) struggle to generate comprehensive comments for dependent methods, a trend also reflected in LLM-based approaches like ASAP. To address this, we propose HelpCOM, a novel dependency-aware technique that incorporates helper method information to improve comment clarity, comprehensiveness, and relevance. Experiments show that HelpCOM outperforms baseline methods by 5.6% to 50.4% across syntactic (e.g., BLEU), semantic (e.g., SentenceBERT), and LLM-based evaluation metrics. A survey of 156 software practitioners further confirms that HelpCOM significantly improves the comprehensibility of code involving dependent methods, highlighting its potential to enhance documentation, maintainability, and developer productivity in large-scale systems.

Figures

Figures reproduced from arXiv: 2504.19459 by the authors.

Figure 1
Figure 1. Motivating example from a real-world project. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A step-by-step overview of our methodology. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Frequency of commit and engagement by method [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative analysis of the generated summaries Between HelpCOM and baselines. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Practitioner understandability of given dependent [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 32 canonical work pages

  1. [1]

    Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang

  2. [2]

    Toufique Ahmed and Premkumar Devanbu. 2022. Multilingual training for soft- ware engineering. In Proceedings of the 44th International Conference on Software Engineering. 1443–1455

    2022

  3. [3]

    Toufique Ahmed, Prem Devanbu, Christoph Treude, and Michael Pradel. 2024. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? ArXiv abs/2408.05534 (2024). https://api.semanticscholar.org/CorpusID:271855028

    arXiv 2024

  4. [4]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code sum- marization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  5. [5]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  6. [6]

    Aakash Bansal, Sakib Haque, and Collin McMillan. 2021. Project-level encoding for neural source code summarization of subroutines. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) . IEEE, 253–264

    2021

  7. [7]

    Md Mustakim Billah, Palash Ranjan Roy, Zadia Codabux, and Banani Roy. 2024. Are Large Language Models a Threat to Programming Platforms? An Exploratory Study. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 2...

    work page doi:10.1145/3674805 2024

  8. [8]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems . 1877– 1901

    2020

Show all 88 references
  1. [9]

    Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering (1994), 528–532

    1994

  2. [10]

    John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun- Hsuan Sung, Brian Strope, and Ray Kurzweil

    Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun- Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder. arXiv:1803.11175 [cs.CL] https://arxiv.org/abs/1...

    2018 arXiv

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  4. [12]

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2018. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. arXiv:1705.02364 [cs.CL] https://arxiv.org/ abs/1705.02364

    2018 arXiv

  5. [13]

    Sergio Cozzetti B de Souza, Nicolas Anquetil, and Káthia M de Oliveira. 2005. A study of the documentation essential to software maintenance. In Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive informatio...

    2005

  6. [14]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022)

    2022 arXiv

  7. [15]

    Brian P Eddy, Jeffrey A Robinson, Nicholas A Kraft, and Jeffrey C Carver. 2013. Evaluating source code summarization techniques: Replication and expansion. In 2013 21st International Conference on Program Comprehension (ICPC) . IEEE, 13–22

    2013

  8. [16]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)

    2020 arXiv

  9. [17]

    Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions 2, 212-236 (1981), 22–23

    1981

  10. [18]

    Beat Fluri, Michael Wursch, and Harald C Gall. 2007. Do code and comments co-evolve? on the relation between source code and comment changes. In 14th Working conference on reverse engineering (WCRE 2007) . IEEE, 70–79

    2007

  11. [19]

    Beat Fluri, Michael Würsch, Emanuel Giger, and Harald C Gall. 2009. Analyzing the co-evolution of comments and source code. Software Quality Journal 17 (2009), 367–394

    2009

  12. [20]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large language models are few- shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference...

    2024

  13. [21]

    Felix Grund, Shaiful Alam Chowdhury, Nick C Bradley, Braxton Hall, and Reid Holmes. 2021. CodeShovel: Constructing method-level source code histories. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1510–1522

    2021

  14. [22]

    Qiuhan Gu. 2023. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 2201–2203

    2023

  15. [23]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020)

    2020 arXiv

  16. [24]

    Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010. Supporting program comprehension with source code summarization. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2 . 223–226

    2010

  17. [25]

    Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010. On the use of automated text summarization techniques for summarizing source code. In 2010 17th Working conference on reverse engineering . IEEE, 35–44

    2010

  18. [26]

    Sakib Haque, Zachary Eberhart, Aakash Bansal, and Collin McMillan. 2022. Se- mantic similarity metrics for evaluating source code summarization. In Proceed- ings of the 30th IEEE/ACM International Conference on Program Comprehension . 36–47

    2022

  19. [27]

    Yoshiki Higo, Shinpei Hayashi, and Shinji Kusumoto. 2020. On tracking Java methods with Git mechanisms. Journal of Systems and Software 165 (2020), 110571

    2020

  20. [28]

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th conference on program comprehension . 200– 210

    2018

  21. [29]

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2020. Deep code comment generation with hybrid lexical and syntactical information. Empirical Software Engineering 25 (2020), 2179–2217

    2020

  22. [30]

    Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018. Summarizing source code with transferred API knowledge. In Proceedings of the 27th Interna- tional Joint Conference on Artificial Intelligence . 2269–2275

    2018

  23. [31]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019)

    2019 arXiv

  24. [32]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Sum- marizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016. Association for Computational Linguistics, 2073–2083

    2016

  25. [33]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  26. [34]

    Madhava Krishna, Bhagesh Gaur, Arsh Verma, and Pankaj Jalote. 2024. Using LLMs in Software Requirements Specifications: An Empirical Evaluation. arXiv preprint arXiv:2404.17842 (2024)

    2024 arXiv

  27. [35]

    Abhishek Kumar, Sonia Haiduc, Partha Pratim Das, and Partha Pratim Chakrabarti. 2024. LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization. ArXiv abs/2409.00630 (2024). https://api.semanticscholar. org/CorpusID:272368407

    2024 arXiv

  28. [36]

    Jahnavi Kumar and Sridhar Chimalakonda. 2024. Code summarization without direct access to code-towards exploring federated llms for software engineering. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 100–109

    2024

  29. [37]

    Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 795–806

    2019

  30. [38]

    Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. 2024. Enhancing Static Analysis for Practical Bug Detection: An LLM-Integrated Approach. Proceedings of the ACM on Programming Languages 8, OOPSLA1 (2024), 474–499

    2024

  31. [39]

    Chen Lin, Zhichao Ouyang, Junqing Zhuang, Jianqiang Chen, Hui Li, and Rongxin Wu. 2021. Improving code summarization with block-wise abstract syntax tree splitting. In 2021 IEEE/ACM 29th International Conference on Program Compre- hension (ICPC). IEEE, 184–195

    2021

  32. [40]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  33. [41]

    Hong Yi Lin, Patanamon Thongtanunam, Christoph Treude, and Wachiraphan Charoenwet. 2024. Improving automated code reviews: Learning from experience. In Proceedings of the 21st International Conference on Mining Software Repositories . 278–283

    2024

  34. [42]

    Mario Linares-Vásquez, Boyang Li, Christopher Vendome, and Denys Poshy- vanyk. 2015. How do developers document database usages in source code?(n). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 36–41

    2015

  35. [43]

    Kaibo Liu, Yiyang Liu, Zhenpeng Chen, Jie M Zhang, Yudong Han, Yun Ma, Ge Li, and Gang Huang. 2024. LLM-Powered Test Case Generation for Detecting Tricky Bugs. arXiv preprint arXiv:2404.10304 (2024)

    2024 arXiv

  36. [44]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. EASE 2025, 17–20 June, 2025, Istanbul, Türkiye Md Mustakim Billah, Md Shamimur Rahman, and Banani Roy arXiv preprint arXiv:2303.1...

    2023 arXiv

  37. [45]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. LLaMA-Reviewer: Ad- vancing code review automation with large language models through parameter- efficient fine-tuning. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 647–658

    2023

  38. [46]

    Minh-Thang Luong. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)

    2015 arXiv

  39. [47]

    Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60

    1947

  40. [48]

    Nuno Marques, Rodrigo Rocha Silva, and Jorge Bernardino. 2024. Using ChatGPT in Software Requirements Engineering: A Comprehensive Review.Future Internet 16, 6 (2024), 180

    2024

  41. [49]

    Robert C Martin. 2009. Clean code: a handbook of agile software craftsmanship . Pearson Education

    2009

  42. [50]

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  43. [51]

    1989.Introduction to the practice of statistics

    David S Moore and George P McCabe. 1989.Introduction to the practice of statistics. WH Freeman/Times Books/Henry Holt & Co

    1989

  44. [52]

    Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. 2023. Developer- intent driven code comment generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 768–780

    2023

  45. [53]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  46. [54]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828 (2023)

    2023 arXiv

  47. [55]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  48. [56]

    Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597 (2022)

    2022 arXiv

  49. [57]

    N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  50. [58]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems . 1–7

    2021

  51. [59]

    Paige Rodeghero, Collin McMillan, Paul W McBurney, Nigel Bosch, and Sidney D’Mello. 2014. Improving automated source code summarization via an eye- tracking study of programmers. In Proceedings of the 36th international conference on Software engineering. 390–401

    2014

  52. [60]

    Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing auto- matic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ...

    2021

  53. [61]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  54. [62]

    Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering 14 (2009), 131–164

    2009

  55. [63]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 (2022)

    2022 arXiv

  56. [64]

    Diomidis Spinellis. 2010. Code documentation. IEEE software 27, 4 (2010), 18–19

    2010

  57. [65]

    Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay- Shanker. 2010. Towards automatically generating summary comments for java methods. In Proceedings of the 25th IEEE/ACM international conference on Auto- mated software engineering. 43–52

    2010

  58. [66]

    Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analy- sis of source code comments. In 2013 21st international conference on program comprehension (icpc). Ieee, 83–92

    2013

  59. [67]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source Code Summarization in the Era of Large Language Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Soc...

    2024

  60. [68]

    I Sutskever. 2014. Sequence to Sequence Learning with Neural Networks. arXiv preprint arXiv:1409.3215 (2014)

    2014 arXiv

  61. [69]

    Ze Tang, Xiaoyu Shen, Chuanyi Li, Jidong Ge, Liguo Huang, Zhelin Zhu, and Bin Luo. 2022. Ast-trans: Code summarization with efficient tree-structured attention. In Proceedings of the 44th International Conference on Software Engineering . 150– 162

    2022

  62. [70]

    Ted Tenny. 1988. Program readability: Procedures versus comments. IEEE Transactions on Software Engineering 14, 9 (1988), 1271–1279

    1988

  63. [71]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  64. [72]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

    2015

  65. [73]

    Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018. Improving automatic source code summarization via deep rein- forcement learning. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering . 397–407

    2018

  66. [74]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048 (2023)

    2023 arXiv

  67. [75]

    Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, Zhou Zhao, Jian Wu, S Yu Philip, and Guandong Xu. 2020. Reinforcement-learning-guided source code sum- marization using hierarchical attention.IEEE Transactions on software Engineering 48, 1 (2020), 102–119

    2020

  68. [76]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922 (2023)

    2023 arXiv

  69. [77]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  70. [78]

    Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2019. A large- scale empirical study on code-comment inconsistencies. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 53–64

    2019

  71. [79]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Break- throughs in statistics: Methodology and distribution . Springer, 196–202

    1992

  72. [80]

    Scott N Woodfield, Hubert E Dunsmore, and Vincent Y Shen. 1981. The effect of modularization and comments on program comprehension. In Proceedings of the 5th international conference on Software engineering . 215–223

    1981

  73. [81]

    Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, and Philip S Yu. 2024. Can Large Language Models Serve as Evaluators for Code Summarization? arXiv preprint arXiv:2412.01333 (2024)

    2024 arXiv

  74. [82]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1482–1494

    2023

  75. [83]

    Chunqiu Steven Xia and Lingming Zhang. 2023. Conversational automated program repair. arXiv preprint arXiv:2301.13246 (2023)

    2023 arXiv

  76. [84]

    Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E Hassan, and Shan- ping Li. 2017. Measuring program comprehension: A large-scale field study with professionals. IEEE Transactions on Software Engineering 44, 10 (2017), 951–976

    2017

  77. [85]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming . 1–10

    2022

  78. [86]

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020. Retrieval-based neural source code summarization. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 1385–1397

    2020

  79. [87]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  80. [2021]

    arXiv preprint arXiv:2103.06333 (2021)

    Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333 (2021)

    2021 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.