Pith. sign in

REVIEW 3 major objections 6 minor 40 references

Evolved natural-language rewrite rules plus error feedback recover up to 30% of LLM coding failures, three to five times more than self-repair alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 08:29 UTC pith:2MDMPGP6

load-bearing objection Practical dual-channel repair that turns 10–30% of hard coding failures into passes with reusable, zero-shot-transferable rewrite rules; the main soft spot is un-audited semantic preservation by the rewriter. the 3 major comments →

arxiv 2607.05121 v1 pith:2MDMPGP6 submitted 2026-07-06 cs.SE

From Failing to Passing: Evolving Natural Language Prompt Optimization Rules for LLM Code Generation

Amal Akli , Melissa Akli , Cedric Richter , Mike Papadakis , Yves Le Traon This is my paper
classification cs.SE
keywords prompt optimizationLLM code generationsearch-based software engineeringautomatic program repairDualFixnatural language transformation rulesLiveCodeBenchAPPS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models often fail coding tasks not only because their code is wrong, but because the problem statement itself is worded in ways that push the model toward the wrong interpretation. This paper shows that short, reusable natural-language transformation rules—discovered by a genetic search that rewrites surface phrasing without adding new facts—can turn many of those failures into passes. Combined with ordinary execution-feedback repair in a staged pipeline called DualFix, the method recovers up to 30% of baseline failures on LiveCodeBench and APPS across three generators, fixing three to five times more cases than simply feeding the model its own error messages. The rules are error-agnostic, reusable across problems, and transfer zero-shot to models they were never evolved on. A sympathetic reader would care because this offers a lightweight, model-independent way to raise coding success rates without fine-tuning or hand-rewriting every prompt.

Core claim

Specification-level wording defects form a substantial and largely disjoint source of LLM code-generation failures. A genetic search (RuleEvol) that evolves compact IF-THEN-EXCEPT rewrite rules can clarify those statements so a frozen model produces correct code; when the rules are staged after one round of error-based rewriting, DualFix recovers up to 30% of baseline failures and fixes 3–5× more cases than Self-Fix. The rules alone already fix 10–30% of failures, including 12–17% that pure execution feedback never solves, and they transfer zero-shot across models.

What carries the argument

DualFix—a staged pipeline that first rewrites a failing problem statement conditioned on its error message, then, if still failing, applies an evolved set of natural-language transformation rules discovered by Pareto genetic search (RuleEvol). The rules are short IF-THEN-EXCEPT patterns that restate surface phrasing without changing semantics; they carry the argument by showing that specification clarification and implementation repair target complementary failure classes.

Load-bearing premise

The method assumes the external rewriter never quietly adds or drops facts when applying rules, so measured gains reflect pure clarification rather than leaked extra information.

What would settle it

If an independent audit of every rewritten prompt shows that a non-negligible fraction introduce new constraints or hints absent from the original statement, and those altered cases account for most of the measured recovery, the claim that pure specification rewriting drives the gains would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Prompt wording itself can be treated as a first-class repair target, complementary to execution feedback.
  • Once evolved, the same compact rule set can be reused across problems and models with no further search.
  • Zero-shot transfer of rules means one model’s optimization can lift another model’s pass rate without re-optimization.
  • Many coding failures that resist multi-round self-repair are actually surface-specification problems.
  • The rules could later be applied as a general pre-filter before any generation, preventing failures rather than only repairing them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same search-for-sensitive-phrasing idea could be applied outside code, for example to math word problems or general reasoning tasks where surface notation misleads models.
  • If the shared core of four clarity patterns (math-to-English, terminology, binary predicates, naming) is truly model-agnostic, a small public rule library might become standard preprocessing for competitive-programming prompts.
  • The 66–83% of failures that remain unsolved may mark a natural ceiling for pure-specification repair and point toward hybrids that also expand model knowledge.
  • Tracking when a rule helps versus regresses could diagnose how much of a model’s coding skill is locked behind surface format rather than algorithmic understanding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DualFix, a staged repair pipeline for LLM code generation that first rewrites a failing problem statement using execution-error feedback and, if that fails, rewrites the original statement with a set of evolved natural-language transformation rules. Rules are discovered offline by RuleEvol, a Pareto-based genetic search that mutates compact IF–THEN–EXCEPT patterns under a pass/fail objective, then selected on a held-out select set. The authors evaluate on LiveCodeBench and APPS with three generators (Qwen2.5-Coder-7B, Codestral-22B, Claude Haiku 4.5), reporting that DualFix recovers up to ~30% of baseline failures (3–5× Self-Fix), that rule-based and error-based channels fix largely disjoint failure sets, and that rules evolved on Codestral transfer zero-shot to Claude while still beating error-only baselines.

Significance. If the results hold under the stated constraints, the work is a solid empirical contribution to LLM code generation and automated program repair. It reframes prompt sensitivity as a dual-level repair problem (specification vs. implementation), supplies reusable error-agnostic rewrite rules rather than per-task system prompts, and demonstrates zero-shot cross-model transfer—an uncommon and valuable result. Strengths include clear baselines (Self-Fix, Error-Based, Iterative-Error), component ablations with overlap analysis (Fig. 3), stratified held-out splits with a separate select set, and an empty-seed control in Threats. The method is practical: rules are short text artifacts applied by a frozen rewriter, and DualFix needs only one failing test. These elements make the paper of interest to the SE and LLM-for-code communities even if some interpretive claims need tightening.

major comments (3)
  1. Abstract vs. §V.B / Fig. 3: the abstract states that evolved transformations fix “12–17% of failures that execution-based repair alone cannot resolve,” but the ablation text reports only 4.5–8.4% of baseline failures fixed exclusively by Rule-Based rewriting (and 0–1.1% exclusively by single-round Error-Based). Please reconcile these ranges with an explicit definition (exclusive set size, DualFix-only set, or another quantity) and align abstract, takeaways, and Fig. 3 so the central “disjoint channels” claim is numerically consistent.
  2. §III.B.1, rule-application prompt, and Construct Validity (§VII): the interpretation that exclusive rule-based gains are pure specification clarification rests on the claim that the Rewriter “never adds new facts, constraints, or hints.” Enforcement is prompt-only; no automatic check or human audit of rewritten statements is reported. Given that GPT-4o-mini is a stronger model than some generators, silent leakage (clarifying parentheticals, expanded constraints, solution-relevant rephrasings) could inflate exclusive rule gains and transfer results. Please add a sample audit (e.g., 30–50 fail-to-pass rewrites) scoring semantic preservation, and report how often the hard constraint is violated; if violations occur, re-estimate exclusive gains on the clean subset.
  3. §VII (Internal validity) and seed rules: the empty-seed run rediscovers core patterns on training data but fails to generalize to the test set, so the six hand-written seeds are load-bearing inductive bias rather than optional warm-start. The paper should state this more clearly in the main results narrative (not only Threats), quantify how much of the final rule set is seed-derived vs. search-discovered, and, if feasible, report one additional seed-ablation (e.g., random or alternative seed set) so readers can judge sensitivity of the 10–30% recovery numbers to initialization.
minor comments (6)
  1. §IV.D / Technical Details: evolutionary hyperparameters (10 generations, Pareto size 7, gate batch 40, max 10 rules, etc.) are fixed without sensitivity analysis. A short appendix table varying generations or gate margin would strengthen reproducibility claims.
  2. Fig. 4 fail-to-pass example is helpful; please also show one case where rules regress a previously passing solution (or state that none occurred on the select/test sets) so readers can assess downside risk of unconditional rule application.
  3. Table IV “universal” rules: the binary-predicate and function-name rules are described only briefly. Listing the exact final rule text for the four universal patterns (as done for the Fig. 4 example) would make transfer results more interpretable and reusable.
  4. Pass@1 under greedy decoding is the sole metric (§VII Construct validity). A brief note on whether temperature sampling or pass@k changes the relative ranking of DualFix vs. Self-Fix would help external validity without requiring a full re-run.
  5. Minor wording: abstract “DUALFIX” / body “DualFix” capitalization is inconsistent; “turns to passing” appears repeatedly and can be replaced by “converts to passing” or “repairs.”
  6. Related Work §II.C: GEPA is cited as the search template; a one-sentence contrast of what DualFix optimizes (instance-level specification rewrites) vs. GEPA/MIPRO (system-level prompts) would sharpen the novelty claim for readers familiar with that line of work.

Circularity Check

0 steps flagged

No circularity: DualFix claims rest on held-out execution oracles, not on predictions forced by construction or load-bearing self-citation.

full rationale

The paper is an empirical search-based software engineering study. RuleEvol evolves IF–THEN–EXCEPT rewrite rules on a development/validation split, selects the final rule set on a held-out select set, and reports acceptance rates only on completely held-out LiveCodeBench and APPS test splits. The metric is pass/fail under a sandboxed execution oracle against the benchmarks’ tests—not a quantity algebraically determined by the fitness objective. DualFix’s staged pipeline (error-based rewrite then rule-based rewrite of the original statement) is evaluated by the same external oracle; Self-Fix, Error-Based, and Iterative-Error ablations are independent baselines, not reparameterizations of the same fit. Self-citations [2],[3] supply motivation about prompt sensitivity and defective task descriptions; they do not underwrite uniqueness, forbid alternatives, or force the reported recovery rates. Seed rules supply inductive bias, but the empty-seed ablation and the external test evaluation keep the central claims non-circular. Semantic-preservation risk for the Rewriter is a construct-validity concern, not a circular reduction of prediction to input. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central empirical claims rest on standard LLM-code-generation assumptions, a small set of search hyper-parameters chosen by the authors, and the invented DualFix/RuleEvol machinery. No physical constants or deep mathematical axioms are involved; the free parameters are ordinary experimental knobs and the invented entities are the methodological constructs themselves.

free parameters (3)
  • evolutionary search hyper-parameters (generations=10, children/parent=2, Pareto size=7, gate batch=40, max merges=10, ru
    Chosen by the authors; acceptance of a child requires margin ≥1 on the gate batch. Different values would alter which rules survive and the final acceptance rates.
  • six hand-written seed rules used for population initialization
    Provide inductive bias; empty-seed ablation rediscovers core patterns but fails to generalize to the test set, so the seeds affect the reported numbers.
  • data-split seed s=42 and 100-problem select/validation holds
    Fixed but arbitrary; different splits could change which rules are selected.
axioms (3)
  • domain assumption LLMs are sufficiently sensitive to surface phrasing of coding problems that localized, semantics-preserving rewrites can convert fail→pass without adding information.
    Stated in the introduction and used as the premise for RuleEvol; supported by cited prompt-sensitivity literature but not proved for the specific rule language.
  • ad hoc to paper The external Rewriter (GPT-4o-mini) correctly applies only matching rules and never injects new constraints or examples.
    Enforced solely by the rule-application prompt; acknowledged as a threat in §VII.
  • domain assumption Pass@1 under greedy decoding on the provided test suites is a valid proxy for “correct code generation.”
    Standard in the code-generation literature the paper cites; incomplete suites or timeouts could misclassify solutions.
invented entities (2)
  • RuleEvol (Pareto-based genetic search over IF-THEN-EXCEPT natural-language transformation rule sets) no independent evidence
    purpose: Discover reusable, error-agnostic rewrite rules that improve downstream code generation.
    Adaptation of GEPA-style reflective evolution to coding specifications; the concrete rule format and mutation operators (reflect/delete/merge) are paper-specific.
  • DualFix staged repair pipeline no independent evidence
    purpose: Compose error-based rewriting (first) with rule-based rewriting (fallback) to address both implementation-level and specification-level failures.
    The ordering and independence of the two channels are design choices introduced by the authors.

pith-pipeline@v1.1.0-grok45 · 21417 in / 3093 out tokens · 27465 ms · 2026-07-11T08:29:50.617164+00:00 · methodology

0 comments
read the original abstract

Large language models are known to be sensitive to prompt formulation. Even minor variations in wording can substantially degrade performance. This sensitivity reveals an opportunity: if prompt phrasing can harm performance, can it be used to improve it? To investigate this question, we introduce a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance. We then propose DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures. A key strength of our approach lies in its generality: the evolved rules are error-agnostic, reusable across problems, and transferable across models. We evaluate DUALFIX against execution-feedback repair baselines across three models on two challenging benchmarks, LiveCodeBench and APPS. Our results show that the evolved transformations fix from 10-30% of failing cases, including 12-17% of failures that execution-based repair alone cannot resolve. Overall, DualFix recovers up to 30% of baseline failures and fixes 3-5 times more failing cases than Self-Fix across all evaluated settings. Furthermore, we also show that rules evolved on one model transfer zero-shot to other models, outperforming execution-feedback repair without any re-optimization.

Figures

Figures reproduced from arXiv: 2607.05121 by Amal Akli, Cedric Richter, Melissa Akli, Mike Papadakis, Yves Le Traon.

Figure 1
Figure 1. Figure 1: The DualFix inference pipeline: Error-based repair is attempted first; [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of RuleEvol. A metaheuristic search optimization loop that evolves transformation rule sets over [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overlap analysis of fixed failures across model-benchmark settings. Each bar shows the proportion of acceptance rate attributed to each approach [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A fail-to-pass example on Codestral-22B. The original problem uses LaTeX markup and presents the task goal only in formal notation. Three evolved [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Validation accuracy of the best rule set across evolutionary generations [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 8 linked inside Pith

  1. [1]

    GEPA: Reflective prompt evolution can outperform reinforcement learning,

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab, “GEPA: Reflective prompt evolution can outperform reinforcement learning,” in The Fourteenth International Conference on Learning Representations, 2026

    2026

  2. [2]

    Defective task descriptions in LLM-based code generation: Detection and analysis,

    A. Akli, M. Papadakis, M. Cordy, and Y . Le Traon, “Defective task descriptions in LLM-based code generation: Detection and analysis,”arXiv preprint arXiv:2604.24703, 2026. [Online]. Available: https://arxiv.org/abs/2604.24703

    Pith/arXiv arXiv 2026

  3. [3]

    When prompt under-specification improves code correctness: An exploratory study of prompt wording and structure effects on LLM-based code generation,

    ——, “When prompt under-specification improves code correctness: An exploratory study of prompt wording and structure effects on LLM-based code generation,”arXiv preprint arXiv:2604.24712, 2026. [Online]. Available: https://arxiv.org/abs/2604.24712

    Pith/arXiv arXiv 2026

  4. [4]

    System card: Claude Haiku 4.5,

    Anthropic, “System card: Claude Haiku 4.5,” Anthropic, Tech. Rep., 2025. [Online]. Available: https://anthropic.com/ claude-haiku-4-5-system-card

    2025

  5. [5]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    POSIX: A prompt sensitivity index for large language models,

    A. Chatterjee, H. S. V . N. S. K. Renduchintala, S. Bhatia, and T. Chakraborty, “POSIX: A prompt sensitivity index for large language models,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12- 16, 2024, ser. Findings of ACL, Y . Al-Onaizan, M. Bansal, and Y . Chen, Eds., vol. EMNLP 2024. Association...

    doi:10.18653/v1/2024.findings-emnlp.852 2024

  7. [7]

    Nlperturbator: Studying the robustness of code llms to natural language variations,

    J. Chen, Z. Li, X. Hu, and X. Xia, “Nlperturbator: Studying the robustness of code llms to natural language variations,”ACM Trans. Softw. Eng. Methodol., vol. 35, no. 4, pp. 89:1–89:20, 2026. [Online]. Available: https://doi.org/10.1145/3745764

    doi:10.1145/3745764 2026

  8. [8]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=KuPixIqPiq

    2024

  10. [10]

    Analyzing prompt influence on automated method generation: An empirical study with Copilot,

    I. D. Fagadau, L. Mariani, D. Micucci, and O. Riganelli, “Analyzing prompt influence on automated method generation: An empirical study with Copilot,” inProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension (ICPC 2024), 2024, pp. 24– 34

    2024

  11. [11]

    Search-based software engineering,

    M. Harman and B. F. Jones, “Search-based software engineering,”Inf. Softw. Technol., vol. 43, no. 14, pp. 833–839, 2001. [Online]. Available: https://doi.org/10.1016/S0950-5849(01)00189-6

    doi:10.1016/s0950-5849(01)00189-6 2001

  12. [12]

    Search-based software engineering: Trends, techniques and applications,

    M. Harman, S. A. Mansouri, and Y . Zhang, “Search-based software engineering: Trends, techniques and applications,”ACM Comput. Surv., vol. 45, no. 1, pp. 11:1–11:61, 2012. [Online]. Available: https://doi.org/10.1145/2379776.2379787

    doi:10.1145/2379776.2379787 2012

  13. [13]

    Measuring coding challenge competence with APPS,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge competence with APPS,” inProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  14. [14]

    Qwen2.5-Coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y . Fan, Y . Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y . Miao, S. Quan, Y . Feng, X. Ren, X. Ren, J. Zhou, and J. Lin, “Qwen2.5-Coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Promptception: How sensitive are large multimodal models to prompts?

    M. I. Ismithdeen, M. U. Khattak, and S. Khan, “Promptception: How sensitive are large multimodal models to prompts?” inFindings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Association for Computational Linguistics, 2025, pp. 23 950–23 98...

    doi:10.18653/v1/2025.findings-emnlp.1302 2025

  16. [16]

    LiveCodeBench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “LiveCodeBench: Holistic and contamination free evaluation of large language models for code,” in International Conference on Learning Representations, 2025

    2025

  17. [17]

    Automated repair of ambiguous problem descriptions for LLM-based code generation,

    H. Jia, R. Morris, H. Ye, F. Sarro, and S. Mechtaev, “Automated repair of ambiguous problem descriptions for LLM-based code generation,” inProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025), 2025. [Online]. Available: https://arxiv.org/abs/2505.07270

    arXiv 2025

  18. [18]

    When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions,

    M. Larbi, A. Akli, M. Papadakis, R. Bouyousfi, M. Cordy, F. Sarro, and Y . Le Traon, “When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions,”arXiv preprint arXiv:2507.20439, 2025. [Online]. Available: https://arxiv.org/abs/2507.20439

    Pith/arXiv arXiv 2025

  19. [19]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, C. Zong, F. Xia, W. Li, and R....

    doi:10.18653/v1/2021.acl-long.353 2021

  20. [20]

    Prompt stability in code LLMs: Measuring sensitivity across emotion- and personality-driven variations,

    W. Ma, Y . Yang, J. Ge, X. Xie, and L. Jiang, “Prompt stability in code LLMs: Measuring sensitivity across emotion- and personality-driven variations,”arXiv preprint arXiv:2509.13680, 2025

    arXiv 2025

  21. [21]

    Self-refine: Iterative refine- ment with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, S. Welleck, B. P. Majumder, S. Gupta, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refine- ment with self-feedback,” inAdvances in Neural Information Processing Systems, vol. 36, 2023

    2023

  22. [22]

    State of what art? A call for multi-prompt LLM evaluation,

    M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky, “State of what art? A call for multi-prompt LLM evaluation,”Trans. Assoc. Comput. Linguistics, vol. 12, pp. 933–949,

  23. [23]

    Available: https://doi.org/10.1162/tacl\ a\ 00681

    [Online]. Available: https://doi.org/10.1162/tacl\ a\ 00681

    doi:10.1162/tacl

  24. [24]

    Is self-repair a silver bullet for code generation?

    T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama, “Is self-repair a silver bullet for code generation?” inInternational Conference on Learning Representations, 2024

    2024

  25. [25]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    OpenAI GPT-5 system card,

    ——, “OpenAI GPT-5 system card,”arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Optimizing instructions and demonstrations for multi- stage language model programs,

    K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab, “Optimizing instructions and demonstrations for multi- stage language model programs,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics, 2024, pp. 9340–9366. [Online...

    2024

  28. [28]

    A multi-language perspective on the robustness of LLM code generation,

    F. Rabbi, Z. Ding, and J. Yang, “A multi-language perspective on the robustness of LLM code generation,”Empir. Softw. Eng., vol. 31, no. 5, p. 146, 2026. [Online]. Available: https://doi.org/10.1007/ s10664-026-10882-8

    2026

  29. [29]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting,

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openr...

    2024

  30. [30]

    Reflex- ion: language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflex- ion: language agents with verbal reinforcement learning,” inAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. ...

    2023

  31. [31]

    Evaluating the zero-shot robustness of instruction-tuned language models,

    J. Sun, C. Shaib, and B. C. Wallace, “Evaluating the zero-shot robustness of instruction-tuned language models,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.net/forum?id=g9diuvxN6D

    2024

  32. [32]

    Mind your format: Towards consistent evaluation of in-context learning improvements,

    A. V oronov, L. Wolf, and M. Ryabinin, “Mind your format: Towards consistent evaluation of in-context learning improvements,” inFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, ser. Findings of ACL, L. Ku, A. Martins, and V . Srikumar, Eds., vol. ACL 2024. Association for Compu...

    doi:10.18653/v1/2024.findings-acl.375 2024

  33. [33]

    ReCode: Robustness evaluation of code generation models,

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V . Kumar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, and B. Xiang, “ReCode: Robustness evaluation of code generation models,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Comp...

    2023

  34. [34]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems, vol. 35, 2022

    2022

  35. [35]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 1482–1494. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00129

    doi:10.1109/icse48619.2023.00129 2023

  36. [36]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, M. Christakis and M. Pradel, Eds. ACM, 2024, pp. 819–831. [Online]. Available: htt...

    doi:10.1145/3650212.3680323 2024

  37. [37]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=Bb4VGOWELI

    2024

  38. [38]

    Large language models are human-level prompt engineers,

    Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: https://openreview.net/forum?id=92gvk82DE-

    2023

  39. [39]

    Prosa: Assessing and understanding the prompt sensitivity of llms,

    J. Zhuo, S. Zhang, X. Fang, H. Duan, D. Lin, and K. Chen, “Prosa: Assessing and understanding the prompt sensitivity of llms,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, ser. Findings of ACL, Y . Al- Onaizan, M. Bansal, and Y . Chen, Eds., vol. EMNLP 2024. Association for Computation...

    doi:10.18653/v1/2024.findings-emnlp.108 2024

  40. [40]

    More than a score: Probing the impact of prompt specificity on LLM code generation,

    Y . Zi, H. Menon, and A. Guha, “More than a score: Probing the impact of prompt specificity on LLM code generation,” inProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2025), 2025, pp. 2380–2402. [Online]. A...

    2025