REVIEW 14 cited by
A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
General large language models (LLMs), represented by ChatGPT, have demonstrated significant potential in tasks such as code generation in software engineering. This has led to the development of specialized LLMs for software engineering, known as Code LLMs. A considerable portion of Code LLMs is derived from general LLMs through model fine-tuning. As a result, Code LLMs are often updated frequently and their performance can be influenced by the base LLMs. However, there is currently a lack of systematic investigation into Code LLMs and their performance. In this study, we conduct a comprehensive survey and analysis of the types of Code LLMs and their differences in performance compared to general LLMs. We aim to address three questions: (1) What LLMs are specifically designed for software engineering tasks, and what is the relationship between these Code LLMs? (2) Do Code LLMs really outperform general LLMs in software engineering tasks? (3) Which LLMs are more proficient in different software engineering tasks? To answer these questions, we first collect relevant literature and work from five major databases and open-source communities, resulting in 134 works for analysis. Next, we categorize the Code LLMs based on their publishers and examine their relationships with general LLMs and among themselves. Furthermore, we investigate the performance differences between general LLMs and Code LLMs in various software engineering tasks to demonstrate the impact of base models and Code LLMs. Finally, we comprehensively maintained the performance of LLMs across multiple mainstream benchmarks to identify the best-performing LLMs for each software engineering task. Our research not only assists developers of Code LLMs in choosing base models for the development of more advanced LLMs but also provides insights for practitioners to better understand key improvement directions for Code LLMs.
Forward citations
Cited by 14 Pith papers
-
JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation
On a new execution-verifiable benchmark, LLMs simplify obfuscated JavaScript far better than rule-based tools but preserve syntax and runtime behavior much less reliably.
-
CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building
An LLM-driven agent, CXXCrafter, automatically builds 587 of 752 C/C++ open-source projects (78%), beating default build commands (39%) and bare LLMs (32 to 38%).
-
ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming
ELABORATION provides a four-stage human-feedback taxonomy and an 8,320-problem dataset, with experiments showing human-LLM collaboration improves pass@1 by about 7 percent.
-
CLEAR: Causal Context-Based Agentic Reasoning for Vulnerability Detection
A multi-agent LLM detector grounded in a causal knowledge graph improves pairwise vulnerable-vs-benign discrimination on PrimeVul and a new Java-Pair dataset.
-
PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents
A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.
-
Plainbook: Data Science, in Plain Language
Plainbook makes data-science notebooks natural-language-first by preserving cell descriptions, generating code via AI, enforcing linear execution via a checkpointing kernel, and adding value-centered cell and global tests.
-
SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
A three-agent locate-suggest-fix framework with a knowledge-graph toolkit resolves 154/300 SWE-Bench-Lite issues with Claude-3.5, outperforming same-model baselines by 5-10 points.
-
Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity
Across 500k Python/Java samples, AI-generated functions are shorter, simpler, and trigger more security findings, while human functions carry more complexity and maintainability warnings.
-
Optimizing Token Choice for Code Watermarking: An RL Approach
An RL-trained policy adaptively biases token choices to watermark LLM-generated code while preserving executable behavior.
-
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.
-
GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion
GRACE combines a multi-level code graph, hybrid text-structure retrieval, and graph fusion to improve repository-level code completion over vanilla and graph-based RAG baselines.
-
SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models
SimuGen uses specialized LLM agents and a Simulink block database to convert diagram images into executable MATLAB/Simulink code, reporting 94.5% average accuracy on nine examples.
-
Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models
A dual-retrieval, threshold-gated, full-parameter fine-tuned code LLM (SelRepair) reports state-of-the-art exact-match patch generation on Tufano and VulRepair benchmarks.
-
An AST-guided LLM Approach for SVRF Code Synthesis
AST-guided fine-tuning of T5 models improves SVRF/DRC code generation accuracy on a private 741-example benchmark, with the best model reaching 62.9% on a custom AST-weighted metric.
Discussion (0). Continue with ORCID to comment.