REVIEW 16 cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR.
Forward citations
Cited by 16 Pith papers
-
ELFuzz: Efficient Input Generation via LLM-driven Synthesis Over Fuzzer Space
ELFuzz automatically evolves LLM-written input generators for large programs, outperforming grammar-based fuzzers in coverage and bug finding on seven benchmarks.
-
MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs
Coordinator-proposer multi-agent repair schedules hunks, proposes candidate patches in parallel, and selects/refines them, fixing 326/835 Defects4J bugs with GPT-3.5 and 420 with Claude-3.5-Sonnet.
-
Towards Automated Formal Verification of zkEVMs Using LLM-Guided Constraint Synthesis
VeriSynth synthesizes executable Z3 verification models from zkEVM Rust source via LLM-guided translation with solver-backed checking, detecting 91.6% of 95 injected opcode bugs versus 46.3% for a direct LLM baseline ...
-
TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management
TokenMizer builds a knowledge graph of LLM sessions and serializes it into 78-token resume blocks that retain more task, decision, and file information than flat-text baselines at roughly half the token cost.
-
SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
A three-agent locate-suggest-fix framework with a knowledge-graph toolkit resolves 154/300 SWE-Bench-Lite issues with Claude-3.5, outperforming same-model baselines by 5-10 points.
-
Neurosymbolic Characterization for Reliable Access Control Policy Analysis
PolicySummarizer generates a simplified, human-readable regular expression for the requests a cloud access-control policy allows, and uses automata-based model counting to check the simplification against the exact po...
-
LLM Performance for Code Generation on Noisy Tasks
LLMs solve heavily obfuscated benchmark tasks, and performance decay under obfuscation differs sharply between old and new datasets, which the authors interpret as a signature of training-data contamination.
-
Development of Automated Software Design Document Review Methods Using Large Language Models
Converting tabular design documents into header-aware Markdown or JSON formats lets GPT models catch cross-document inconsistencies with recall up to 0.96 on short documents, but performance collapses beyond 5000 characters.
-
Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness
Feeding static-analysis warnings back to GPT-4o over ten iterations cut security issues from over 40% to about 13%, readability issues from over 80% to about 11%, and reliability warnings from over 50% to about 11%.
-
Augmenting the Generality and Performance of Large Language Models for Software Engineering
A proposal for augmenting LLM generality in non-code software engineering tasks, with no reported experimental evidence.
-
Symphony: A Decentralized Multi-Agent Framework for Scalable Collective Intelligence
Symphony's decentralized multi-agent LLM framework claims strong accuracy gains but its evaluation has internal contradictions and missing statistical support.
-
The role of large language models in UI/UX design: A systematic literature review
A systematic review of 38 studies finds LLMs are increasingly integrated across the UI/UX design lifecycle, with prompt engineering and human oversight as key practices.
-
Evaluating the Performance and Efficiency of Sentence-BERT for Code Comment Classification
On the NLBSE 2025 code comment classification task, small Sentence-BERT models with tuned classifiers improve F1 by +0.0346 with only 1.4x runtime and 2.1x GFLOPS overhead.
-
Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects
GPT-4 detected command injection in popular Python projects with 75.5% accuracy and F1 74.5%, missing 15 real cases and beating Bandit on precision.
-
Knowledge Graph Based Repository-Level Code Generation
A knowledge graph code retrieval pipeline is described, but its headline results come from an evaluation that skips the retrieval step and anchors context on the known target function.
-
A Comprehensive Study on the Use of Word Embedding Models in Software Engineering Domain
A systematic review of 181 papers shows word embeddings are most used for software maintenance and development, with Word2Vec and BERT as the dominant models and weak evidence for their advantage over simpler baselines.
Discussion (0). Sign in to comment.