REVIEW 11 cited by
Source Code Summarization in the Era of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
To support software developers in understanding and maintaining programs, various automatic (source) code summarization techniques have been proposed to generate a concise natural language summary (i.e., comment) for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of code-related tasks. In this paper, we undertake a systematic and comprehensive study on code summarization in the era of LLMs, which covers multiple aspects involved in the workflow of LLM-based code summarization. Specifically, we begin by examining prevalent automated evaluation methods for assessing the quality of summaries generated by LLMs and find that the results of the GPT-4 evaluation method are most closely aligned with human evaluation. Then, we explore the effectiveness of five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert) in adapting LLMs to code summarization tasks. Contrary to expectations, advanced prompting techniques may not outperform simple zero-shot prompting. Next, we investigate the impact of LLMs' model settings (including top\_p and temperature parameters) on the quality of generated summaries. We find the impact of the two parameters on summary quality varies by the base LLM and programming language, but their impacts are similar. Moreover, we canvass LLMs' abilities to summarize code snippets in distinct types of programming languages. The results reveal that LLMs perform suboptimally when summarizing code written in logic programming languages compared to other language types. Finally, we unexpectedly find that CodeLlama-Instruct with 7B parameters can outperform advanced GPT-4 in generating summaries describing code implementation details and asserting code properties. We hope that our findings can provide a comprehensive understanding of code summarization in the era of LLMs.
Forward citations
Cited by 11 Pith papers
-
Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles
Vul4Py gives 100 real Python vulnerabilities a paired exploit-and-functional oracle, and on this benchmark agentic repair systems outperform direct prompting and a specialized tool by a wide margin.
-
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Prompt language affects LLM code generation, but English is not consistently best: Chinese prompts improve Python correctness on CoderEval, while quality and lexicon effects vary by model and programming language.
-
Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization
The best token-pruning strategy for code summarization is language-dependent: AST inputs help Java but hurt Python, while function signatures are best for Python.
-
UserTrace: User-Level Requirements Generation and Traceability Recovery from Software Project Repositories
UserTrace generates user-level requirements from code repositories and recovers live trace links from requirements to implementation, with evaluations suggesting gains over summarization and traceability baselines.
-
Positional Biases Shift as Inputs Approach Context Window Limits
The lost-in-the-middle effect in LLMs appears mainly when inputs fill up to half the model's context window; beyond that, accuracy favors information closest to the end.
-
When Retriever Meets Generator: A Joint Model for Code Comment Generation
RAGSum couples retriever and generator in one CodeT5 model with contrastive pre-training, joint fine-tuning, and ROUGE-L based self-refinement, outperforming CMR-Sum, JOINTCOM, and Llama-3.1-8B on three code-comment datasets.
-
Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?
Filtering code summarization training data by code-comment coherence gives no better results than random selection, and halving the training set does not hurt performance.
-
Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code
Mutation-based testing shows LLM-generated code summaries frequently fail to reflect behavior-changing edits, with accuracy dropping as code complexity rises.
-
Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks
Multi-task QLoRA on Qwen2.5-Coder matches or beats single-task QLoRA and full fine-tuning for code generation and Python summarization, but lags in Java-to-C# translation.
-
Enhancing Semantic Understanding in Pointer Analysis using Large Language Models
LMPA is a proposed, unevaluated framework for using LLMs to improve pointer analysis through API abstraction, parameter inference, and natural-language summaries.
-
Enhancing COBOL Code Explanations: A Multi-Agents Approach Using Large Language Models
A two-agent LLM pipeline with hierarchical merging explains COBOL functions, files, and projects, outperforming zero-shot baselines on several text-quality metrics.
Discussion (0). Continue with ORCID to comment.