Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-agent LLM pipeline can generate end-to-end documentation for legacy COBOL codebases.

desk verdict Solid, honest engineering for a neglected problem; needs an ablation and a few wording fixes before it is fully convincing. read the letter →

arxiv 2507.02182 v1 pith:J7JAOZEV submitted 2025-07-02 cs.SE

Fangjian Lei , Jiawen Liu , Shayan Noei , Ying Zou , Derek Truong , William Alexander This is my paper
classification cs.SE
keywords COBOLcodeexplanationmulti-agentLLMartifactshierarchicalmergingLLM-as-a-judgelegacysoftwaredocumentationlong-contextlimitation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a two-agent large-language-model pipeline can generate accurate explanations for COBOL code at three levels: individual functions, whole files, and entire projects, without manual documentation effort. The motivation is that COBOL still powers much of the financial and government infrastructure but often lacks current documentation and has a shrinking pool of experienced developers. The authors claim that pairing a code-trained model with a text-refining model, and feeding both models code artifacts such as variable names, called functions, and file dependencies, produces explanations that outperform a zero-shot baseline at every level. They further claim that splitting long files into functions and hierarchically merging their explanations lets the pipeline explain COBOL files that exceed an LLM's token window. If right, this would let organizations auto-generate current documentation for legacy COBOL systems and, by extension, for other languages with long source files.

What carries the argument

The carrying mechanism is a two-agent pipeline built on hierarchical merging: a code-processing LLM (granite-34b-code-instruct) generates preliminary explanations from source code plus artifacts, and a text-processing LLM (GPT-4o-mini) refines and merges those explanations in a bottom-up pass. Functions that call no others keep their preliminary explanation; each caller's explanation is then rewritten with the final explanations of its callees substituted in, and the same merge is applied upward to files and then to the project using dependency graphs. This lets the system explain inputs larger than the code model's 8,192-token window by never presenting a whole oversized file at once.

What would settle it

Have an independent team, blind to the system's origin and to the developer comments, write fresh behavioral descriptions for a sample of functions and files from the 14 projects, then rerun the function-, file-, and project-level comparisons against those independent descriptions; if the two-agent pipeline no longer beats the zero-shot baseline, the reference-comment assumption is the reason.

Watch

Extended reading notes

Core claim

The central claim is that explanation quality for COBOL improves when code understanding and text polishing are separated into two collaborating agents, with source-code artifacts supplied at each granularity. The Code Processing Agent reads the code and artifacts and writes preliminary explanations; the Text Processing Agent merges and rewrites them, at the function level by substituting called-function explanations for bare call names, at the file level by merging function explanations along dependency relationships, and at the project level by merging file explanations along a topological file-dependency graph. The authors report that this approach beats a zero-shot baseline by 12.69% in METEOR, 18.59% in chrF, and 0.62% in SentenceBERT at the function level, and that their LLM-as-a-judge evaluation favors it by 6.01 to 5.57 out of 10. At the file level, it improves purpose, functionality, and clarity by 4.21%, 10.72%, and 14.68% respectively, including for files longer than the model's token window; at the project level, generated descriptions match or surpass existing READMEs in 9 of 11 projects, with manual checks confirming the project purpose in most cases.

Load-bearing premise

The evaluation assumes that the original developer comments selected as the reference dataset are a valid ground truth for what the code is supposed to do, so that similarity to those comments and judge ratings against them actually measure explanation quality.

Editorial extensions

If this is right

  • Organizations can generate function-, file-, and project-level documentation for COBOL codebases without relying on developers to write or update comments.
  • Files that exceed an LLM's token window can still be explained, because the pipeline segments code and merges lower-level explanations hierarchically.
  • Incorporating code artifacts such as variable names, called functions, and dependency relationships into prompts is what drives the gain over zero-shot prompting.
  • Generated project explanations can serve as an alternative to README files for projects with missing or outdated documentation.
  • The artifact-extraction and hierarchical-merging techniques are intended to generalize to other programming languages, though the experiments in this paper are COBOL-only.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reference-comment assumption holds, a similar two-agent split could help other under-documented legacy languages where files are long and comments are stale.
  • The reported functionality scores near 1.0 for files suggest the bottleneck is not code understanding but the quality of the reference descriptions; a harder test would compare explanations against independently written behavioral specs rather than developer comments.
  • Because file- and project-level explanations are built from function-level explanations, errors can propagate upward; a fault-injection study that perturbs one low-level explanation and traces the effect upward would quantify that risk.
  • The paper reports quality improvements but not the added latency or token cost of running two models, so a cost-benefit extension would measure whether the gains justify the extra LLM calls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a multi-agent pipeline for generating COBOL code explanations at function, file, and project levels. A Code Processing Agent (Granite-34b-code-instruct) analyzes source code, while a Text Processing Agent (GPT-4o-mini) refines and hierarchically merges explanations, using extracted artifacts such as variable names, called functions, and file/project dependencies. The approach is evaluated on 14 open-source COBOL projects against a zero-shot baseline using text-similarity metrics (METEOR, chrF, SentenceBERT), an LLM-as-a-Judge, and manual evaluation by the authors. The reported results include improvements of 12.69%, 18.59%, and 0.62% at the function level, file-level improvements in purpose, functionality, and clarity, and project-level explanations judged to match or surpass README files in 9 of 11 tested projects.

Significance. If the central claims are supported, this paper addresses a genuinely underserved problem: generating documentation for legacy COBOL systems. The hierarchical merging strategy for long files is a practical contribution that may transfer to other languages whose code exceeds LLM context windows. The authors also provide a replication package, evaluate on real-world open-source projects, and include statistical tests with effect sizes, which is commendable. However, the evaluation design does not currently isolate the contribution of the proposed components, and the reference dataset may measure alignment with a small, terse set of retained original comments rather than explanation usefulness. These issues need to be resolved before the headline claims can be accepted.

major comments (3)
  1. [Abstract and Section 4 (RQ1), Table 1] The abstract states that the approach 'performs significantly better than the baseline in function code explanation, with improvements of 12.67%, 18.59%, and 0.62% in terms of METEOR, chrF, and SentenceBERT scores.' This grouping is misleading: Table 1 marks the SentenceBERT improvement as 'not significant' in the Signif. Diff row, and the METEOR improvement is significant only with a negligible effect size. Please either restrict the 'significantly better' claim to the metrics that are statistically significant, or report the significance status and effect sizes alongside each metric in the abstract and RQ1 summary.
  2. [Section 4 (RQ1 Approach) and Section 3.5.1] The RQ1 baseline is a zero-shot prompt to the Code Processing Agent with no additional artifacts. The proposed function-level pipeline differs from this baseline in two ways: it adds artifact prompts (variables, called functions) and it adds a second LLM, the Text Processing Agent, which refines and merges the Code Processing Agent's outputs. The conclusion in Section 4 that 'Providing code artifacts enhances the performance of LLMs' is therefore not supported by the comparison as designed, because the two factors are varied jointly. An ablation study (for example, artifacts without text-agent refinement, and text-agent refinement without artifacts) is needed to attribute the observed gains to the stated cause.
  3. [Section 3.2 and Section 4 (RQ1-RQ3)] The evaluation of the central claim is anchored to a reference dataset built by filtering original developer comments, but this same reference is used as the gold standard in every evaluation channel: the text-similarity metrics, the LLM-as-a-Judge benchmark, and the manual reference-coverage scoring. Section 3.2 reports that only 243 of 2,316 functions (10.5%) survived the authors' filter, and Section 3.5.1 states that the retained function comments have a median length of 10 words, while generated explanations are prompted to be about 75 words. Consequently, the metrics largely measure whether generated text contains the content of a small set of terse comments, not whether the explanations are correct or useful for the majority of functions whose comments were filtered out. The manual verification is also performed by the same two authors who constructed the reference set, so it does not break this loop. The paper should either obtain independent human judgments (for example, annotators blind to the original comments, or a developer-usefulness study on a sample that is not restricted to the filtered subset) or explicitly limit the claims to 'alignment with retained original comments.'
minor comments (6)
  1. [Abstract and Section 4 (RQ1)] The METEOR improvement is reported as 12.67% in the abstract and 12.69% in Section 4; please make these consistent.
  2. [Section 3.4.2] The text reads 'we use use GPT-4o-mini'; the duplicated word should be corrected.
  3. [Section 3.5.1] The description of the 75-word threshold does not state what data or evaluation criterion was used in the 'gradient search' (e.g., which explanations, which metric, whether this was done before or during the main evaluation). Please document this procedure so readers can assess the risk of threshold overfitting.
  4. [Section 4 (RQ2), Table 3] The 'Signif. Diff' row for Clarity reports four asterisks (****), which does not match the significance codes defined in the table footnote. Please correct the notation.
  5. [Abstract and Section 4 (RQ3)] The project-level claim of '82% of the selected projects' is based on 9 of 11 projects that have reference READMEs, not 9 of the 14 selected projects. Please clarify the denominator and explain why only 11 projects are included in this evaluation.
  6. [Section 4 (RQ1), Manual Verification] The manual verification reports a median reference-coverage score of 1 and a mean of 0.773; the text describes the results as closely aligned but does not report the mean or the spread. Please report both mean and median for each manual metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-agent pipeline is not fitted to its evaluation reference, and load-bearing citations are external.

full rationale

The paper's derivation chain is an empirical pipeline: extract code artifacts, run two LLM agents (Granite-34b-code-instruct and GPT-4o-mini) with prompt templates, hierarchically merge explanations, then compare outputs against a reference dataset built from original developer comments and README files. At no point does a parameter, prompt, or explanation get fit to the evaluation metrics (METEOR, chrF, SentenceBERT, or the GPT-4o judge). The 75-word output threshold is described as chosen by a gradient search over length thresholds, but this is a prompt-verbosity hyperparameter selected before the reported evaluation, not a quantity derived from the test labels; the headline improvements are relative to a zero-shot baseline under the same reference data, so they are not forced by construction. The reference dataset is external to the generation pipeline: it consists of pre-existing developer comments and README files filtered by two annotators, and the generated explanations never receive the reference as input during function-, file-, or project-level generation. The manual 'reference coverage' evaluation is an operationalization of alignment with those comments, not a re-derivation of the output from the metric. The paper does cite prior work by co-authors (e.g., [34] for an 80% language-threshold criterion), but that citation is not load-bearing for the central claim. The hierarchical merging technique is attributed to external work ([52], [11]), not to a self-citation chain, and no uniqueness theorem is imported from the authors' own prior papers. Concerns about using terse, filtered developer comments as ground truth are correctness and threat-to-validity issues, not circularity: a flawed gold standard weakens the conclusions but does not make the comparison reduce to its inputs. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on these assumptions: ground-truth comments are valid, the judge and metrics are valid proxies, merging summaries loses no essential information, and the project sample is representative. None of these are verified beyond the paper's own small validation, so they are listed as axioms.

free parameters (1)
  • 75-word explanation threshold = 75 words
    Chosen by gradient search over 50, 75, 100, 200 words in Section 3.5.1 to balance brevity and detail; a prompt-formatting hyperparameter, not fitted to the evaluation metrics.
assumptions (6)
  • domain assumption The developer comments used as reference are accurate ground truth for code intent
    The reference dataset (Section 3.2) is built from original developer comments, filtered by two authors, and used as the benchmark for similarity metrics and LLM-judge; if comments are stale or incomplete, the evaluation targets the wrong quantity.
  • domain assumption LLM-as-a-judge (GPT-4o) gives valid quality scores
    Section 3.4.3 assumes GPT-4o can judge clarity, conciseness, correctness, purpose, and functionality; only a small manual validation (54 functions, 72 files) is provided, so the judge's reliability is assumed beyond that sample.
  • domain assumption Text similarity metrics (METEOR, chrF, SentenceBERT) measure explanation quality
    Used in RQ1 as alignment with reference; these are lexical/semantic overlap measures, not necessarily correlated with developer usefulness.
  • domain assumption Hierarchical merging preserves information needed for file/project explanation
    The long-file and project pipelines (Sections 3.5.2, 3.5.3) assume that merging function explanations recovers file/project-level meaning; this is a known assumption from book summarization (Wu et al. 2021, Booookscore).
  • ad hoc to paper The selected 14 projects represent COBOL practice
    The dataset is 7/14 IBM projects plus a public COBOL dataset; the paper acknowledges this dominance in Section 6 as a threat to generalizability.
  • domain assumption Granite-34b-code-instruct is trained on COBOL code explanation tasks
    Section 3.4.1 asserts this capability, attributing the model's effectiveness to COBOL-specific training; no COBOL-specific benchmark is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Models." pith.science (2026). https://pith.science/paper/J7JAOZEV

@misc{pith2026250702182,
  author       = {Pith},
  title        = {Pith review of: Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7JAOZEV}},
  note         = {Machine review of arXiv:2507.02182}
}
read the original abstract

Common Business Oriented Language (COBOL) is a programming language used to develop business applications that are widely adopted by financial, business, and government agencies. Due to its age, complexity, and declining number of COBOL developers, maintaining COBOL codebases is becoming increasingly challenging. In particular, the lack of documentation makes it difficult for new developers to effectively understand and maintain COBOL systems. Existing research utilizes large language models (LLMs) to explain the functionality of code snippets. However, COBOL presents unique challenges due to its architectural and syntactical differences, which often cause its code to exceed the token window size of LLMs. In this work, we propose a multi-agent approach that leverages two LLM-based agents working collaboratively to generate explanations for functions, files, and the overall project. These agents incorporate together by utilizing contextual information from the codebase into the code explanation prompts. We evaluate the effectiveness of our approach using 14 open-source, real-world COBOL projects. Our results indicate that our approach performs significantly better than the baseline in function code explanation, with improvements of 12.67%, 18.59%, and 0.62% in terms of METEOR, chrF, and SentenceBERT scores, respectively. At the file level, our approach effectively explains both short and long COBOL files that exceed the token window size of LLMs and surpass the baseline by 4.21%, 10.72%, and 14.68% in explaining the purpose, functionality, and clarity of the generated explanation. At the project level, our approach generates explanations that convey the functionality and purpose of 82% of the selected projects.

Figures

Figures reproduced from arXiv: 2507.02182 by the authors.

Figure 1
Figure 1. Example of COBOL program that accumulates an [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An Overview of Our Multi-agent Approach on Generating Code Explanation at Function, File and Project Levels. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of pairing code processing with [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging

    cs.SE 2026-07 conditional novelty 6.0 of 10

    SEDCoT combines LLM translation, symbolic-execution test generation, and delta-debugging repair to raise COBOL-to-C correctness by ≥12% over SOTA LLM baselines while preserving human-readable output.

Reference graph

Works this paper leans on

55 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documenta- tion issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1199–1210

    work page 2019

  2. [2]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code sum- marization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    work page 2024

  3. [3]

    V Aho Alfred, S Lam Monica, and D Ullman Jeffrey. 2007. Compilers principles, techniques & tools. pearson Education

    work page 2007

  4. [4]

    Mir Sameed Ali, Nikhil Manjunath, and Sridhar Chimalakonda. 2022. COBREX: A Tool for Extracting Business Rules from COBOL. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 464–468

    work page 2022

  5. [5]

    Mir Sameed Ali, Nikhil Manjunath, and Sridhar Chimalakonda. 2023. X-cobol: A dataset of cobol repositories. arXiv preprint arXiv:2306.04892 (2023)

    work page Pith review arXiv 2023

  6. [6]

    Anonymous. 2025. Enhancing COBOL Code Explanations: A Multi-Agents Ap- proach Using Large Language Models. https://github.com/anonymous-987654321/ ICSE2026 Accessed: 2025-03-14

    work page 2025

  7. [7]

    Anthropic. 2023. The Claude 3 Model Family: Opus, Sonnet, Haiku. https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf Accessed: 18 May 2024

    work page 2023

  8. [8]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

Show all 55 references
  1. [9]

    Paheli Bhattacharya, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh

  2. [10]

    Håkan Burden and Rogardt Heldal. 2011. Natural language generation from class diagrams. In Proceedings of the 8th International Workshop on Model-Driven Engineering, Verification and Validation. 1–8

    2011

  3. [11]

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2023. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785 (2023)

    2023 arXiv

  4. [12]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv preprint arXiv:2310.14735 (2023)

    2023 arXiv

  5. [13]

    Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li

  6. [14]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  7. [15]

    Agnieszka Ciborowska, Aleksandar Chakarov, and Rahul Pandita. 2021. Contem- porary COBOL: Developers’ perspectives on defects and defect location. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 227–238

    2021

  8. [16]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46

    1960

  9. [17]

    Anh TV Dau, Hieu Trung Dao, Anh Tuan Nguyen, Hieu Trung Tran, Phong X Nguyen, and Nghi DQ Bui. 2024. XMainframe: A Large Language Model for Mainframe Modernization. arXiv preprint arXiv:2408.04660 (2024)

    2024 arXiv

  10. [18]

    Sergio Cozzetti B de Souza, Nicolas Anquetil, and Káthia M de Oliveira. 2005. A study of the documentation essential to software maintenance. In Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive informatio...

    2005

  11. [19]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 (2023)

    2023 arXiv

  12. [20]

    Golara Garousi, Vahid Garousi-Yusifoğlu, Guenther Ruhe, Junji Zhi, Mahmoud Moussavi, and Brian Smith. 2015. Usage and usefulness of technical software documentation: An industrial case study. Information and software technology 57 (2015), 664–682

    2015

  13. [21]

    Aric Hagberg, Pieter J Swart, and Daniel A Schult. 2008. Exploring network structure, dynamics, and function using NetworkX . Technical Report. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

    2008

  14. [22]

    Nipuni T Hewage. 2024. Automatic Migration of Java Platform Threads to Virtual Threads. (2024)

    2024

  15. [23]

    Raisa Islam and Owana Marzia Moushi. 2024. Gpt-4o: The cutting-edge advance- ment in multimodal llm. Authorea Preprints (2024)

    2024

  16. [24]

    Thomas Junk. 1999. Confidence level computation for combining searches with small statistics. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 434, 2-3 (1999), 435–443

    1999

  17. [25]

    Charles Kiefer. 2017. COBOL as a modern language

    2017

  18. [26]

    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Sci...

    2023

  19. [27]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634 (2023)

    2023 arXiv

  20. [28]

    Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the gpt-3 large language model. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2. 37–39

    2022

  21. [29]

    McBurney

    Paul W. McBurney. 2015. Automatic Documentation Generation via Source Code Summarization. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. 903–906. doi:10.1109/ICSE.2015.288

    2015 doi

  22. [30]

    Paul W McBurney and Collin McMillan. 2014. Automatic documentation gen- eration via source code summarization of method context. In Proceedings of the 22nd International Conference on Program Comprehension . 279–290

    2014

  23. [31]

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. 2024. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:...

    2024 arXiv

  24. [33]

    Nadim Nachar et al. 2008. The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in quantitative Methods for Psychology 4, 1 (2008), 13–20

    2008

  25. [34]

    Shayan Noei, Heng Li, Stefanos Georgiou, and Ying Zou. 2023. An Empirical Study of Refactoring Rhythms and Tactics in the Software Development Process. IEEE Transactions on Software Engineering 49, 12 (2023), 5103–5119

    2023

  26. [35]

    Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation . 392–395

    2015

  27. [36]

    Gede Artha Azriadi Prana, Denae Ford, Ayushi Rastogi, David Lo, Rahul Pu- randare, and Nachiappan Nagappan. 2021. Including everyone, everywhere: Understanding opportunities and challenges of geographic gender-inclusion in oss. IEEE Transactions on Software Engineering 48, 9 (...

    2021

  28. [37]

    Vaclav Rajlich. 1997. Comprehension and evolution of legacy software (tutorial). In Proceedings of the 19th international conference on Software engineering . 669– 670

    1997

  29. [38]

    N Reimers. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  30. [39]

    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 27–43. Conference ac...

    2022

  31. [40]

    Mamillapally Raghavender Sharma. 2020. A short communication on computer programming languages in modern era. Int. J. Comput. Sci. Mob. Comput 9 (2020), 50–60

    2020

  32. [41]

    Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay- Shanker. 2010. Towards automatically generating summary comments for java methods. In Proceedings of the 25th IEEE/ACM international conference on Auto- mated software engineering. 43–52

    2010

  33. [42]

    Giriprasad Sridhara, Lori Pollock, and K Vijay-Shanker. 2011. Automatically detecting and describing high level actions within methods. In Proceedings of the 33rd International Conference on Software Engineering . 101–110

    2011

  34. [43]

    Giriprasad Sridhara, Lori Pollock, and K Vijay-Shanker. 2011. Generating pa- rameter comments and integrating with method summaries. In 2011 IEEE 19th international conference on program comprehension . IEEE, 71–80

    2011

  35. [44]

    Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analy- sis of source code comments. In 2013 21st International Conference on Program Comprehension (ICPC). 83–92. doi:10.1109/ICPC.2013.6613836

    2013

  36. [45]

    Chia-Yi Su and Collin McMillan. 2024. Distilled GPT for source code summariza- tion. Automated Software Engineering 31, 1 (2024), 22

    2024

  37. [46]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source code summarization in the era of large language models. arXiv preprint arXiv:2407.07959 (2024)

    2024 arXiv

  38. [47]

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784 (2024)

    2024 arXiv

  39. [48]

    Tom Taulli. 2020. COBOL language: Call it a comeback? Retrieved January 13 (2020), 2022

    2020

  40. [49]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  41. [50]

    Ashish Upadhaya. 2023. Understanding Legacy Software: The Current Relevance of COBOL. (2023)

    2023

  42. [51]

    Akhila Sri Manasa Venigalla and Sridhar Chimalakonda. 2022. An Empirical Study On Correlation between Readme Content and Project Popularity. arXiv preprint arXiv:2206.10772 (2022)

    2022 arXiv

  43. [52]

    Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 (2021)

    2021 arXiv

  44. [53]

    Juan Zhai, Jianjun Huang, Shiqing Ma, Xiangyu Zhang, Lin Tan, Jianhua Zhao, and Feng Qin. 2016. Automatic model generation from documentation for Java API functions. In Proceedings of the 38th International Conference on Software Engineering. 380–391

    2016

  45. [54]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges.arXiv preprint arXiv:2310.17631 (2023)

    2023 arXiv

  46. [2022]

    Informa- tion Processing & Management 59, 6 (2022), 103069

    Legal case document similarity: You need both network and text. Informa- tion Processing & Management 59, 6 (2022), 103069

    2022

  47. [2023]

    In International Conference on Artificial Intelligence in Education

    GPTutor: a ChatGPT-powered programming tool for code explanation. In International Conference on Artificial Intelligence in Education . Springer, 321–327

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.