REVIEW 7 cited by
Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recently, the ChatGPT LLM has received great attention: it can be used as a bot for discussing source code, prompting it to suggest changes, provide descriptions or even generate code. Typical demonstrations generally focus on existing benchmarks, which may have been used in model training (i.e., data leakage). To assess the feasibility of using an LLM as a useful assistant bot for programmers, we must assess its realistic capabilities on unseen problems as well as its capabilities on various tasks. In this paper, we present an empirical study of ChatGPT's potential as a fully automated programming assistant, focusing on the tasks of code generation, program repair, and code summariziation. The study investigates ChatGPT's performance on common programming problems and compares it with state-of-the-art approaches on two benchmarks. Among several findings, our study shows that ChatGPT is effective in dealing with common programming problems. However, our experiments also reveal limitations in terms of its attention span: detailed descriptions will constrain the focus of ChatGPT and prevent it from leveraging its vast knowledge to solve the actual problem. Surprisingly, we have identified the ability of ChatGPT to reason the original intention of the code. We expect future work to build on this insight for dealing with the open question of the oracle problem. Our findings contribute interesting insights to the development of LLMs for programming assistance, notably by demonstrating the importance of prompt engineering, and providing a better understanding of ChatGPT's practical applications for software engineering.
Forward citations
Cited by 7 Pith papers
-
Can LLMs Test Terminal User Interfaces?
Automated TUI testing is feasible but unsolved: random input wins on raw crash counts under a time budget, LLM guidance wins per interaction and on input-gated faults, and line coverage does not predict crash finding.
-
AgenTag: Attribution of AI Coding Agents from Behavioral Fingerprints
A multimodal, contrastive-learning system attributes GitHub PRs to five AI coding agents and humans with weighted F1 0.96, and shows that text style, not code, carries the attribution signal.
-
AdaptAgent: A Multi-agent, Domain-Guided Reasoning Framework for Code Adaptation
A multi-agent LLM pipeline that plans code adaptations using summarized intent, domain checklists, and sibling-method context outperforms single-shot prompting and repair baselines on Java adaptation examples.
-
AI-Assisted Modeling: DSL-Driven AI Interactions
A conversational, voice-enabled modeling workflow for the Lingua Franca DSL with instant diagram feedback and grammar-steered LLM tool calls.
-
On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
Even the best tested LLM judge, GPT-4-turbo, frequently misjudges code correctness (e.g., 50% of wrong Java functions judged correct) but agrees moderately with humans when judging code summary quality.
-
When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning
LLM-based sketch generation for embedded ML is fragile, with success rates below 40%, and prompt structure alone can swing outcomes from 15% to 30%.
-
Lossless Token Sequence Compression via Meta-Tokens
A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.
Discussion (0). Continue with ORCID to comment.