Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

AuPair: Golden Example Pairs for Code Repair

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Providing a different curated (guess, fix) example on every repair call beats best-of-N and self-repair.

desk verdict Solid empirical paper with a genuinely new example-selection mechanism; the core claim holds up, but missing seeds/artifacts and an untested Phase-1 coverage assumption keep it short of a clean accept. read the letter →

arxiv 2502.18487 v1 pith:Z6NPIFOL submitted 2025-02-12 cs.SE cs.AIcs.CLcs.LG

Aditi Mavalankar , Hassan Mansoor , Zita Marinho , Masha Samsikova , Tom Schaul This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords coderepairin-contextlearninginference-timecomputesubmodularselectioncompetitiveprogrammingself-repairfew-shotpromptingLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that the best use of extra inference-time compute for code repair is not to sample more fixes independently or to ask the model to critique itself, but to give the model a different worked example of repairing a wrong guess on each call. It presents an algorithm that synthesises many (guess, fix) pairs, then greedily selects an ordered subset that are individually useful and jointly complementary, and feeds one selected pair as a one-shot in-context example per repair attempt. On seven competitive programming datasets and five LLMs, the authors report that this AuPair prompting beats best-of-32 sampling and verbal-feedback self-repair, generalises out of distribution, and keeps improving with more compute where baselines plateau. If correct, the result means curated few-shot repair examples are a cheap, model-agnostic lever for inference-time scaling.

What carries the argument

The central object is the AuPair: a (guess, fix) pair together with the guess's and fix's unit-test scores, used as a one-shot in-context example. The carrying mechanism is the submodular extraction procedure: build a fix-quality matrix $M$ in which entry $M_{ij}$ is the unit-test score achieved when pair $c_i$ repairs validation problem $x_j$; iteratively append the pair with the highest row mean to the ordered AuPair list, then subtract that row, clipped to $[0,1]$, from all rows so subsequent selections focus on problems still unsolved. This is what makes the final ordered set complementary by construction.

What would settle it

Take a test set made of problems whose dominant bugs were deliberately excluded from the Phase 1 repair budget; if AuPair then matches rather than beats best-of-$N$ at $N=32$, the claim that selection creates the advantage is falsified. Alternatively, a scaling run past 144 AuPairs that shows a plateau while best-of-$N$ continues climbing would contradict the reported scaling advantage.

Watch

Extended reading notes

Core claim

The central claim is that a submodularly selected, ordered set of golden example pairs—each containing an initial buggy guess, an improved fix, and their unit-test scores—acts as a much stronger repair prompt than any single fixed prompt or self-generated feedback, when each LLM call gets the next AuPair as a one-shot example. The algorithm's selection step computes a fix-quality matrix over a validation set, repeatedly picks the pair with the highest mean fix score, and subtracts that pair's row so the next choice is rewarded for solving problems the previous pairs could not. The authors report that with $N$ calls the best of the $N$ repaired solutions passes more tests than best-of-$N$ and self-repair at matching budgets, with 2.5–3x compute savings over randomly chosen pairs and a scaling curve that does not plateau at 144 calls.

Load-bearing premise

The method assumes Phase 1 collects at least one useful (guess, fix) pair for every bug class that appears in test problems, since the selection phase can only retrieve what the bounded candidate pool contains; if a common failure mode is never repaired during pair generation, AuPair gains nothing over baselines on problems with that failure mode.

Editorial extensions

If this is right

  • On the two in-distribution datasets, CodeForces and AtCoder, AuPair outperforms best-of-$N$ and self-repair on all five models at a fixed $N=32$ budget, sometimes by wide margins.
  • AuPairs are 2.5–3x more compute-efficient than randomly selected repair pairs: 12 AuPairs reach the score that 32 random pairs reach.
  • The test pass rate rises roughly log-linearly with $N$ for at least 144 AuPairs on CodeForces with Gemini-1.5-Pro, while best-of-$N$ and self-repair plateau.
  • AuPairs extracted on one dataset (CodeForces) transfer to six other datasets, matching or exceeding in-distribution AuPairs on smaller datasets such as LiveCodeBench.
  • AuPairs transfer across models: fixes generated with a stronger repair model benefit more from high-quality AuPairs, while the source model matters little for Gemini-1.5-Flash.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: because Phase 1 only seeds the pool, any existing corpus of buggy-and-fixed code pairs could replace it; the selection phase should carry most of the benefit, which is testable by running Phase 2 alone on external paired data.
  • Editorial: the method's reliance on unit-test scores suggests it should transfer to other verifiable domains, such as formal proofs or data-processing scripts, but ungrounded reward-model feedback would need new validation, as the authors note.
  • Editorial: the monotone ordering of AuPairs means a user can stop early—the first pairs give the largest gains—which is useful for latency-constrained applications.
  • Editorial: a direct stress test would be to vary the Phase 1 budget per bug class and measure whether performance on that class tracks coverage in the candidate pool, making the paper's weakest assumption explicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AuPair, a two-phase method for improving LLM code repair at inference time. Phase 1 generates candidate (guess, fix) pairs by iteratively repairing initial solutions on a training set with k-shot prompts that draw randomly from the growing candidate pool. Phase 2 evaluates each candidate pair as a 1-shot example on a validation set, builds a fix-quality matrix, and applies a greedy submodular selection procedure (Algorithm 2) that subtracts the selected pair's scores from the remaining rows to obtain an ordered, complementary list of AuPairs. At inference, for a budget of N LLM calls, the first N AuPairs are used as in-context examples to produce N repaired solutions, and the highest-scoring solution on the unit tests is selected. The paper reports experiments on 5 LLMs and 7 competitive programming datasets, comparing test pass rate and strict accuracy against best-of-N and self-repair, with additional ablations on random pairs, out-of-distribution datasets, cross-model transfer, difficulty levels, and problem categories.

Significance. If the central claims hold, the paper makes a useful empirical contribution: a training-free method that converts a precomputed set of repair exemplars into a complementary ordered list, with consistently large gains over best-of-N and self-repair and better scaling with the number of inference calls. The evaluation is unusually broad—5 models, 7 datasets, plus out-of-distribution and cross-model axes—and the random-pair ablation in §3.2 is a good step toward isolating the value of selection. The illustrative AuPair examples in §A.9 and the lineage analysis in §A.5 are informative. However, the main claims are empirical and currently lack statistical error bars, an offline-cost accounting for the two phases, and a repeated-best-pair baseline; these gaps need to be addressed before the compute-efficiency and complementarity claims are fully supported.

major comments (4)
  1. [§2.2 (Algorithm 1) and §3.3] The paper's central compute-efficiency claim compares only the per-problem inference budget N, but Phase 2 requires evaluating every candidate pair against every validation problem. For Gemini-1.5-Pro on CodeForces this is roughly 1,560 pairs times approximately 1,100 validation problems, or about 1.7M LLM calls, none of which appear in Fig. 5(b) or in the scaling comparison. Please report the total LLM-call cost of Phase 1 and Phase 2, or clearly state that the comparison is amortized over many test problems; otherwise 'more efficient use of compute' is not established.
  2. [§3.2 (Fig. 5(a))] The ablation compares AuPairs with randomly selected pairs, but it does not include the natural control of using the single best-scoring pair (by validation row-mean) in all N inference calls. Without this control, the gain over random pairs could reflect the quality of the first AuPair rather than the complementarity of the ordered set. Please add this repeated-best-pair baseline; it is directly implied by the claim that 'selection matters' and that complementarity is the mechanism.
  3. [§A.5 and Table 2] The method's mechanism assumes that the candidate pool C contains a useful (guess, fix) example for each error type that will appear at test time. The paper's own lineage analysis states that fix counts decay with depth and that 'several problems could not be improved beyond a certain point, or that they were not resampled,' and the generated pools are small (e.g., 147 pairs for Gemma-9B on AtCoder). The category analysis in Fig. 8 tracks problem tags, not repair archetypes, so it does not establish bug-type coverage. Please report per-problem coverage, such as the fraction of test problems whose initial guess is improved by at least one candidate pair, and how this coverage varies with the Phase-1 budget. Without this, the risk that the reported averages hide a large class of problems for which the method degenerates to unguided repair remains open.
  4. [§3.1 and Fig. 4] All headline results are single-run point estimates, and the text repeatedly uses 'significant' without a statistical test. Because LLM sampling at temperature 1.0 is stochastic, please report multiple seeds or bootstrap confidence intervals over test problems, and a paired significance test for the main AuPair-versus-best-of-N and AuPair-versus-self-repair comparisons. This is needed to support the claim that AuPair 'reliably outperforms' baselines rather than merely having a large point estimate.
minor comments (5)
  1. [§3.7, Table 1] The sentence 'our results with Gemini-1.5-Pro indicate improved performance with higher difficulty' appears to contradict the table, which shows decreasing test pass rates from 0.62 (level A) to 0.35 (level F+). Please correct either the text or the table.
  2. [Algorithm 2] The tolerance epsilon is never given a value in the experiments. Please report it and state how sensitive the number of AuPairs is to this choice.
  3. [§3, Baselines] Please state the sampling temperature and decoding parameters used for AuPair inference; best-of-N is set to temperature 1.0, and the comparison is only fair if the same parameters are used for all AuPair runs.
  4. [§A.6, Algorithm 4] Line 11 sets S_max to argmax_i S_diff_i (a set), while the text and Eq. (1) treat S_max as the maximum cardinality. Please make the normalization consistent, for example S_max = max_i |S_diff_i|.
  5. [Fig. 14] The caption says 'the set of all pairs' while the text refers specifically to pairs generated on CodeForces using Gemini-Pro-1.5; please clarify which model and dataset each panel corresponds to and add axis labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AuPairs are selected on a held-out validation set and evaluated on held-out test sets, so the reported gains are empirical rather than definitional.

full rationale

The paper's derivation chain is empirical rather than definitional. Phase 1 (Algorithm 3) generates candidate (guess, fix) pairs from training problems; Phase 2 (Algorithms 1 and 2) computes a fix-quality matrix on the held-out validation set D_val and greedily selects an ordered AuPair list by validation performance. The reported test pass rates are measured on D_test, a disjoint split, using the same prompting and scoring procedure. No equation defines the test metric in terms of the selection objective: the matrix M is built from D_val, while the headline numbers are averages over D_test. The scaling plots evaluate the precomputed ordered list at increasing N; they are empirical curves, not identities forced by the greedy rule. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via a citation; related-work references are contextual. The paper itself flags a genuine coverage limitation in §A.5, noting that fix counts decay with depth and that 'several problems could not be improved beyond a certain point, or that they were not resampled during the pair generation phase.' That is a threat to general coverage, not a circular step. The random-pair ablation is a legitimate control: comparing submodular selection with random selection from the same candidate set tests whether the selection mechanism adds value, and it is an empirical comparison rather than a restatement of the method's definition. Overall, the AuPair construction is validation-set tuning of in-context examples; the test and out-of-distribution results are genuine held-out measurements, so there is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or formal entities. It relies on standard empirical assumptions about LLM in-context learning and unit-test-based correctness, plus a small set of hand-chosen hyperparameters (epsilon, k, phase-1 budget, N, temperature). The most consequential choices for the central claim are epsilon and the phase-1 budget, since they determine the size and composition of the AuPair list.

free parameters (5)
  • epsilon (tolerance in Algorithm 2) = not reported
    Stops AuPair extraction when the best mean fix score falls below epsilon; determines the number of AuPairs. The paper never states its value, so the reported AuPair counts cannot be reproduced exactly.
  • k (number of few-shot examples during pair generation) = 32
    Chosen for diversity during Phase 1; no sensitivity analysis is given.
  • Phase 1 LLM call budget = 35,000 (CodeForces), 10,000 (AtCoder)
    Sets the size of the candidate pair pool; no ablation on how this budget affects downstream performance.
  • Inference compute budget N = 32 for main results; up to 144/110 for scaling
    The user-chosen number of repair attempts; the method's advantage over baselines may depend on this budget.
  • Sampling temperature during fix generation = not reported
    Not specified for repair/fix generation; best-of-N uses 1.0, but the temperature for AuPair inference is unstated, affecting reproducibility.
assumptions (4)
  • domain assumption LLMs can learn to repair code from a single in-context (guess, fix) example
    The entire method relies on few-shot repair ability; assumed in Sections 2 and 3.
  • domain assumption Unit test pass rate is a valid measure of code correctness for selection and evaluation
    Used to score guesses and fixes throughout; the paper acknowledges it only measures pass rate, not other quality aspects.
  • domain assumption The validation set is representative of the test distribution for the same dataset
    AuPair selection uses validation fix-quality scores to pick examples; if validation errors differ systematically from test errors, selection would not transfer.
  • ad hoc to paper The greedy submodular update M <- clip(M - M_k, 0, 1) yields a near-optimal complementary set
    No approximation guarantees are stated; the procedure is a heuristic adopted from submodular maximization, and its effectiveness is only demonstrated empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AuPair: Golden Example Pairs for Code Repair." pith.science (2026). https://pith.science/paper/Z6NPIFOL

@misc{pith2026250218487,
  author       = {Pith},
  title        = {Pith review of: AuPair: Golden Example Pairs for Code Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6NPIFOL}},
  note         = {Machine review of arXiv:2502.18487}
}
abstract

Scaling up inference-time compute has proven to be a valuable strategy in improving the performance of Large Language Models (LLMs) without fine-tuning. An important task that can benefit from additional inference-time compute is self-repair; given an initial flawed response, or guess, the LLM corrects its own mistake and produces an improved response, or fix. We leverage the in-context learning ability of LLMs to perform self-repair in the coding domain. The key contribution of our paper is an approach that synthesises and selects an ordered set of golden example pairs, or AuPairs, of these initial guesses and subsequent fixes for the corresponding problems. Each such AuPair is provided as a single in-context example at inference time to generate a repaired solution. For an inference-time compute budget of $N$ LLM calls per problem, $N$ AuPairs are used to generate $N$ repaired solutions, out of which the highest-scoring solution is selected as the final answer. The underlying intuition is that if the LLM is given a different example of fixing an incorrect guess each time, it can subsequently generate a diverse set of repaired solutions. Our algorithm selects these AuPairs in a manner that maximises complementarity and usefulness. We demonstrate the results of our algorithm on 5 LLMs across 7 competitive programming datasets for the code repair task. Our algorithm yields a significant boost in performance compared to best-of-$N$ and self-repair, and also exhibits strong generalisation across datasets and models. Moreover, our approach shows significantly stronger scaling with inference-time compute budget compared to baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReCode: Improving LLM-based Code Repair with Fine-Grained Retrieval-Augmented Generation

    cs.SE 2025-09 conditional novelty 6.0 of 10

    ReCode improves LLM-based code repair by combining algorithm-type prediction with dual-encoder retrieval of authentic bug-fix pairs, and introduces RACodeBench for evaluation.

Reference graph

Works this paper leans on

44 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwal, A

    R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, L. Rosias, S. Chan, B. Zhang, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle. Many-shot in-context learning, 2024. URL https://arxiv.org/abs/2404.11018

    arXiv 2024

  2. [2]

    Aky \"u rek, D

    E. Aky \"u rek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=0g0X4H8yN4I

    work page 2023

  3. [3]

    Berabi, J

    B. Berabi, J. He, V. Raychev, and M. Vechev. Tfix: Learning to fix coding errors with a text-to-text transformer. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 780--791. PMLR, 18--24 Jul 2021. URL https://proceedings.mlr.press/v139/bera...

    work page 2021

  4. [4]

    Bhatia and R

    S. Bhatia and R. Singh. Automated correction for syntax errors in programming assignments using recurrent neural networks. CoRR, abs/1603.06129, 2016. URL http://arxiv.org/abs/1603.06129

    arXiv 2016

  5. [5]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert - Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    arXiv 2005

  6. [6]

    Chakraborty, Y

    S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray. Codit: Code editing with tree-based neural models. IEEE Transactions on Software Engineering, 48 0 (4): 0 1385–1399, Apr. 2022. ISSN 2326-3881. doi:10.1109/tse.2020.3020502. URL http://dx.doi.org/10.1109/TSE.2020.3020502

    work page arXiv 2022

  7. [7]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. In arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    X. Chen, M. Lin, N. Sch \"a rli, and D. Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023

    arXiv 2023

Show all 44 references
  1. [9]

    Z. Chen, S. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus. Sequencer: Sequence-to-sequence learning for end-to-end program repair. CoRR, abs/1901.01808, 2019. URL http://arxiv.org/abs/1901.01808

    1901 arXiv

  2. [10]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Au...

    2024

  3. [11]

    Devlin, J

    J. Devlin, J. Uesato, R. Singh, and P. Kohli. Semantic code repair using neuro-symbolic transformation networks. CoRR, abs/1710.11054, 2017. URL http://arxiv.org/abs/1710.11054

    2017 arXiv

  4. [12]

    Dinella, H

    E. Dinella, H. Dai, Z. Li, M. Naik, L. Song, and K. Wang. Hoppity: Learning graph transformations to ddetect and fix bugs in programs. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJeqs6EFvB

    2020

  5. [13]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. C ode BERT : A pre-trained model for programming and natural languages. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP...

    2020 doi

  6. [14]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In International Conference on Learning Representations, 2023

    2023

  7. [15]

    Z. Gou, Z. Shao, Y. Gong, yelong shen, Y. Yang, N. Duan, and W. Chen. CRITIC : Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Sx038qxjek

    2024

  8. [16]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps. In Advances in Neural Information Processing Systems, 2021

    2021

  9. [17]

    Y. Hu, X. Shi, Q. Zhou, and L. Pike. Fix bugs with transformer through a neural-symbolic edit grammar, 2022. URL https://arxiv.org/abs/2204.06643

    2022 arXiv

  10. [18]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  11. [19]

    Jiang, T

    N. Jiang, T. Lutellier, and L. Tan. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, May 2021. doi:10.1109/icse43902.2021.00107. URL http://dx.doi.org/10.1109/ICSE4390...

    2021

  12. [20]

    H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems, 2022

    2022

  13. [21]

    R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi...

    2023 arXiv

  14. [23]

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Koh...

    2022 doi

  15. [24]

    Nijkamp, B

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations, 2023

    2023

  16. [25]

    T. X. Olausson, J. P. Inala, C. Wang, J. Gao, and A. Solar-Lezama. Is self-repair a silver bullet for code generation? In International Conference on Learning Representations, 2024

    2024

  17. [26]

    Ouyang, J

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang. An empirical study of the non-determinism of chatgpt in code generation. ACM Trans. Softw. Eng. Methodol., Sept. 2024. ISSN 1049-331X. doi:10.1145/3697010. URL https://doi.org/10.1145/3697010

    2024 doi

  18. [27]

    Renze and E

    M. Renze and E. Guven. The effect of sampling temperature on problem solving in large language models, 2024. URL https://arxiv.org/abs/2402.05201

    2024 arXiv

  19. [28]

    Romera-Paredes, M

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, A. Fawzi, J. Grochow, A. Lodi, J.-B. Mouret, T. Ringer, and T. Yu. Mathematical discoveries from program search with large language mode...

    2023

  20. [29]

    Shirafuji, Y

    A. Shirafuji, Y. Oda, J. Suzuki, M. Morishita, and Y. Watanobe. Refactoring programs using large language models with few-shot examples. In 2023 30th Asia-Pacific Software Engineering Conference (APSEC). IEEE, Dec. 2023. doi:10.1109/apsec60848.2023.00025. URL http://dx.doi.org...

    2023

  21. [30]

    Shypula, A

    A. Shypula, A. Madaan, Y. Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh. Learning performance-improving code edits, 2024. URL https://arxiv.org/abs/2302.07867

    2024 arXiv

  22. [31]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https://arxiv.org/abs/2009.01325

    2009 arXiv

  23. [32]

    Von Oswald, E

    J. Von Oswald, E. Niklasson, E. Randazzo, J. a. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR, 2023

    2023

  24. [33]

    H. Wang, Z. Liu, S. Wang, G. Cui, N. Ding, Z. Liu, and G. Yu. Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing. CoRR, abs/2311.09868, 2023 a . URL http://dblp.uni-trier.de/db/journals/corr/corr2311.html#abs-2311-09868

    2023 arXiv

  25. [34]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models, 2023 b . URL https://arxiv.org/abs/2203.11171

    2023 arXiv

  26. [35]

    Welleck, X

    S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hH36JeQZDaO

    2023

  27. [36]

    J. J. Wu and F. H. Fard. Benchmarking the communication competence of code generation for llms and llm agent. arXiv preprint arXiv:2406.00215, 2024

    2024 arXiv

  28. [37]

    C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, page...

    2022

  29. [38]

    Yasunaga and P

    M. Yasunaga and P. Liang. Break-it-fix-it: Unsupervised learning for program repair. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11941--11952. PMLR, 18--2...

    2021

  30. [39]

    X. Yin, C. Ni, S. Wang, Z. Li, L. Zeng, and X. Yang. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1274–1286, New York, NY, USA, 2024. Association for Compu...

    2024

  31. [40]

    W. Yuan, Q. Zhang, T. He, C. Fang, N. Q. V. Hung, X. Hao, and H. Yin. Circle: continual repair across programming languages. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, page 678–690, New York, NY, USA, 2022. Asso...

    2022

  32. [41]

    Yuan and W

    Y. Yuan and W. Banzhaf. Arja: Automated repair of java programs via multi-objective genetic programming. IEEE Transactions on Software Engineering, 46: 0 1040--1067, 2017. URL https://api.semanticscholar.org/CorpusID:25222219

    2017

  33. [42]

    Y. Zhao, Z. Huang, Y. Ma, R. Li, K. Zhang, H. Jiang, Q. Liu, L. Zhu, and Y. Su. R e P air: Automated program repair with process-based feedback. In Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics, Aug. 2024. URL htt...

    2024

  34. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [45]

    hѢ7o^̟?? .\ k׮dž b׮]ǧe˖? _]vq q'9 ,Y ӧOE ŢEbժUʊ= Dٲev u͚5 NMz ΋/)lٲ+֯_ f͊E Ųebo RJ ׭[7 6l u) ٳ'/_ko|綾(U7>PbůQJ7

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.