Pith. sign in

REVIEW 10 cited by

Is Self-Repair a Silver Bullet for Code Generation?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.09896 v5 pith:37763PWR submitted 2023-06-16 cs.CL cs.AIcs.PLcs.SE

Theo X. Olausson , Jeevana Priya Inala , Chenglong Wang , Jianfeng Gao , Armando Solar-Lezama This is my paper · ORCID
classification cs.CLcs.AIcs.PLcs.SE
keywords self-repaircodefeedbackmodelperformanceabilityboostgains
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Solver-Aware Decompositions for Programming-by-Example: When Dividing Requires Knowing how to Conquer

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Solver-aware training of a PBE decomposer with a frozen synthesizer's loss outperforms supervised imitation of ground-truth subgoals, solving tasks that a ground-truth decomposition oracle fails.

  2. NKI-Agent: Domain-Specific Fine-Tuning and Agentic Tool Use for Neuron Kernel Generation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An agent with compile/verify tools reaches 77.3% NKI kernel pass rate on real Trn1 hardware with Opus 4.8, versus 6% single-shot; SFT Qwen3-30B hits 25% at ~1/100th cost, and binary-reward GRPO fails to beat SFT.

  3. Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

    cs.SE 2026-05 conditional novelty 6.0 of 10

    On IaC-Eval v2, active retrieval raises Qwen 7B Terraform pass@1 from 14% to 45.7%, iterative refinement reaches 84.4% for GPT-4o, and 79% of residual OPA policy failures stem from information gaps.

  4. ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation

    cs.SE 2025-09 conditional novelty 6.0 of 10

    ReCode improves LLM-based code repair by combining algorithm-type prediction with dual-encoder retrieval of authentic bug-fix pairs, and introduces RACodeBench for evaluation.

  5. Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A bootstrapping algorithm that trains program synthesis models on their own successful outputs and on repaired failures modestly improves pass@k over regular fine-tuning on MBPP, with mixed results on APPS.

  6. ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ICPC-Eval benchmarks LLMs on 118 recent ICPC contest problems with local test suites and a Refine@K metric that quantifies iterative refinement, showing even the best models solve only about 29% of problems.

  7. Knowledge-Enhanced Program Repair for Data Science Code

    cs.SE 2025-02 conditional novelty 6.0 of 10

    DSrepair combines a knowledge graph of data science APIs with AST-level bug localization to repair LLM-generated code, fixing more DS-1000 tasks than five baseline repair methods.

  8. Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.

  9. ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle

    cs.CY 2025-07 conditional novelty 5.0 of 10

    Fine-tuning a 7B code model on real student submissions produces code that better matches student error patterns, style, and incremental revision trajectories than prompting-only models across two temporal resolutions.

  10. VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries

    cs.CR 2025-05 reject novelty 4.0 of 10

    An LLM pipeline that enriches decompiled code with vulnerability hints and then classifies CWEs reports high Juliet accuracy, but leaky hints and unmatched baselines weaken the claim.

Pith tools