Pith. sign in

REVIEW 2 cited by

An Analysis of the Automatic Bug Fixing Performance of ChatGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.08653 v1 pith:YX5YXCWA submitted 2023-01-20 cs.SE

Dominik Sobania , Martin Briesch , Carol Hanna , Justyna Petke This is my paper · ORCID
classification cs.SE
keywords chatgptfixingapproachesrepairperformanceprogrammethodssoftware
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. gigiProfiler: Diagnosing Performance Issues by Uncovering Application Resource Bottlenecks

    cs.PF 2025-07 conditional novelty 6.0 of 10

    gigiProfiler combines an LLM resource spotter with static validation to find application-level bottlenecks and root-cause variables by comparing buggy and normal executions.

  2. Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education

    cs.CY 2025-05 conditional novelty 6.0 of 10

    Across four LLMs, pitfall recall in machine-learning code averaged under 50%, with information-leakage and model-selection errors most often missed.

Pith tools