REVIEW 6 cited by
Automatic Code Summarization via ChatGPT: How Far Are We?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
To support software developers in understanding and maintaining programs, various automatic code summarization techniques have been proposed to generate a concise natural language comment for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of natural language processing tasks. Among them, ChatGPT is the most popular one which has attracted wide attention from the software engineering community. However, it still remains unclear how ChatGPT performs in (automatic) code summarization. Therefore, in this paper, we focus on evaluating ChatGPT on a widely-used Python dataset called CSN-Python and comparing it with several state-of-the-art (SOTA) code summarization models. Specifically, we first explore an appropriate prompt to guide ChatGPT to generate in-distribution comments. Then, we use such a prompt to ask ChatGPT to generate comments for all code snippets in the CSN-Python test set. We adopt three widely-used metrics (including BLEU, METEOR, and ROUGE-L) to measure the quality of the comments generated by ChatGPT and SOTA models (including NCS, CodeBERT, and CodeT5). The experimental results show that in terms of BLEU and ROUGE-L, ChatGPT's code summarization performance is significantly worse than all three SOTA models. We also present some cases and discuss the advantages and disadvantages of ChatGPT in code summarization. Based on the findings, we outline several open challenges and opportunities in ChatGPT-based code summarization.
Forward citations
Cited by 6 Pith papers
-
UserTrace: User-Level Requirements Generation and Traceability Recovery from Software Project Repositories
UserTrace generates user-level requirements from code repositories and recovers live trace links from requirements to implementation, with evaluations suggesting gains over summarization and traceability baselines.
-
When Retriever Meets Generator: A Joint Model for Code Comment Generation
RAGSum couples retriever and generator in one CodeT5 model with contrastive pre-training, joint fine-tuning, and ROUGE-L based self-refinement, outperforming CMR-Sum, JOINTCOM, and Llama-3.1-8B on three code-comment datasets.
-
Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebases
A multi-agent bottom-up framework (summarizer + keyword extractor + QA) improves hierarchical code-summary consistency by ~8% and normalized keyword coverage by up to 38% over structured prompting baselines.
-
ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts
An empirical study of developer-ChatGPT refactoring chats yields a 25-theme taxonomy, apology/affirmation signals, and a structured prompt template that reduces conversation turns.
-
Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models
A dual-retrieval, threshold-gated, full-parameter fine-tuned code LLM (SelRepair) reports state-of-the-art exact-match patch generation on Tufano and VulRepair benchmarks.
-
Resource-Efficient & Effective Code Summarization
QLoRA fine-tuning outperforms full fine-tuning for code summarization on CodeXGLUE while cutting GPU memory by roughly two-thirds.
Discussion (0). Continue with ORCID to comment.