Pith. sign in

REVIEW 6 major objections 6 minor 33 references

$T^3$: Multi-level Tree-based Automatic Program Repair with Large Language Models

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read T3, a four-stage tree-search pipeline, repairs up to 48.2% of benchmark bugs, beating chain-of-thought by 9.9–11.2 points.

desk verdict A plausible prompt-level APR pipeline with consistent point-estimate gains, but the evaluation protocol is under-specified enough to block full verification of the headline margins. read the letter →

arxiv 2506.21211 v1 pith:NBK3VTJI submitted 2025-06-26 cs.SE cs.AI

Quanming Liu , Xupeng Bu , Zhichao Yan , Ru Li This is my paper
classification cs.SEcs.AI
keywords automaticprogramrepairlargelanguagemodelschain-of-thoughtpromptingforest-of-thoughtself-consistencyBM25retrievalMODITbenchmarkpatchgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

T3 is a proposed automatic program repair framework that replaces single-path chain-of-thought reasoning with a staged pipeline: retrieve similar code with BM25, analyze error causes through a forest of parallel reasoning trees, generate repair plans from the top-ranked causes, and finally produce a patch. On the MODIT benchmark, the paper reports repair rates of 46.70% (B2Fs) and 28.20% (B2Fm) with gpt-3.5-turbo, and 48.20% (B2Fs) and 32.10% (B2Fm) with gpt-4o-mini; against the gpt-3.5-turbo chain-of-thought baseline, the gains are 11.20 and 9.90 percentage points. The practical point is that staged tree search can improve LLM repair ability without fine-tuning and transfers across models. Ablations are used to argue that both the cause-analysis and repair-planning stages contribute, and that the full multi-level pipeline is needed for the largest gains.

What carries the argument

The load-bearing mechanism is the 'Forest of Thinking': a set of M independent reasoning trees sharing a common root, each expanded by chain-of-thought steps, with outputs aggregated by exact-match frequency. In cause analysis, the forest produces a ranked top-n set of recurring error causes; in repair planning, new trees are rooted in each selected cause and again aggregated by frequency. This frequency-based voting is what turns parallel exploration into a single repair plan, and the paper credits it with balancing diversity and consistency. The final patch is generated by feeding the chosen plan back through the language model with the original program and retrieved examples.

What would settle it

Run T3 and the CoT baseline on the same MODIT subset under the paper's settings (30 generated samples per program, temperature 0.7) with an explicit execution oracle—compile each patch and run the repository's tests—and compare repair rates; if the 11-point gap disappears or reverses, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that decomposing LLM-based repair into multiple tree-structured reasoning stages, rather than one chain, raises the fraction of programs that are repaired. In its 'Forest of Thinking', each of M reasoning trees expands by chain-of-thought, and candidate error causes and repair plans are ranked by how frequently they recur across the forest; the top-ranked items are passed to the next stage. The paper reports that this design beats CoT, Tree-of-Thought, Plan-and-Solve, and Analogical Reasoning on both MODIT subsets under two different LLMs, and that the full pipeline outperforms versions with cause analysis or planning removed. The paper interprets the result as evidence that path diversity, not a stronger single prompt, is what improves repair accuracy.

Load-bearing premise

The load-bearing premise is that the reported repair rate counts genuine fixes, but the paper never states whether a patch must pass hidden tests, compile, or exactly match the gold patch.

Editorial extensions

If this is right

  • If the reported rates hold, a 10–11 point repair gain over chain-of-thought is available from prompting and retrieval alone, with no fine-tuning.
  • The ablation results attribute the gain to both stages: removing repair planning lowers B2Fs repair from 46.70% to 37.60%, and removing cause analysis lowers it to 37.30%.
  • The two-model comparison indicates the gain is not tied to one LLM: gpt-3.5-turbo and gpt-4o-mini both improve, and gpt-4o-mini reaches the highest overall rates (48.20% B2Fs, 32.10% B2Fm).
  • The few-shot analysis shows T3 plateaus at three retrieved examples and beats Plan-and-Solve even with one example, so the improvement is attributed to structure rather than sample quantity.
  • On B2Fm, T3's repair rate is higher than CoT's even though its own self-consistency (SC) accuracy is lower, which the paper takes as evidence that diverse reasoning paths can improve repair without consistent majority voting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the size of the reported 11-point gain depends on the correctness oracle, so the result is best read as a comparison of LLM-generated outputs until a concrete oracle (hidden tests, compilation, or gold-patch matching) is specified.
  • We infer that the same forest-of-thought aggregation could transfer to other code reasoning tasks, such as root-cause localization or test-suite generation, because the mechanism does not depend on repair-specific prompts; the paper does not test these settings.
  • We infer from the B2Fm SC-accuracy reversal that diversity of reasoning paths, rather than majority-vote consistency, may be the active ingredient; a controlled study varying the number of trees while holding total samples fixed could test that mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes T3, a four-stage automatic program repair (APR) pipeline that retrieves similar code with BM25, performs Forest-of-Thinking cause analysis over M parallel reasoning trees, generates repair plans over another forest, and produces patches with an LLM. It evaluates T3 on the MODIT B2Fs and B2Fm datasets using gpt-3.5-turbo and gpt-4o-mini, comparing against CoT, Tree-of-Thought, Plan-and-Solve, Analogical Reasoning, and previously reported S-C and S-C+BM25 results. The headline claims are a 46.70% repair rate on B2Fs and 28.20% on B2Fm with gpt-3.5-turbo, exceeding CoT by 11.20 and 9.90 percentage points, with larger gains on gpt-4o-mini. The paper also presents an ablation study, a shot-count analysis, and a case study. The core idea is plausible, but the empirical protocol is under-specified in ways that affect the central claim.

Significance. If the reported numbers are reproducible, T3 would be a simple, training-free improvement over single-path prompting for APR, and the analysis of sample-count saturation would be practically useful. The paper gets credit for using a standard benchmark, comparing multiple baselines, reporting exact model snapshots, and including ablation and case-study evidence. However, because the correctness oracle and sample-to-label aggregation are unspecified, and key hyperparameters are absent, the significance is conditional; the result cannot currently be verified or compared with prior work on equal footing.

major comments (6)
  1. [Section IV.C, Eq. (15)] The repair rate R = N_correct/N_total is undefined without a correctness oracle. Section IV.B states that each program is tested with 30 generated samples, but it never specifies whether a program counts as repaired when any of the 30 samples passes hidden tests, when a single designated sample passes, or when the majority-vote patch passes; neither does it say whether correctness means passing test cases, compiling, or matching the gold patch. Since the headline margins in Table I and the contribution statement in Section I rest entirely on this metric, the protocol must be specified before the central claim can be interpreted or reproduced.
  2. [Sections III.B-III.C and IV.B] The method is not reproducible because the key hyperparameters are never reported: M (number of trees), N_i (candidates per tree), k (BM25 retrieval count), and top-n (number of selected causes and plans) appear in Eqs. (3), (4), (7), (9), (10), and (13) but no values are given anywhere, including in the ablation and shot-count experiments. Without these values, and without the prompt templates (which are also omitted), the reader cannot re-run the experiments or assess how sensitive the 11.20 and 9.90 percentage-point gains are to these choices.
  3. [Section IV.B and Table I] The table compares T3 and the CoT variants, which use gpt-3.5-turbo/gpt-4o-mini, with the S-C and S-C+BM25 rows, which Section IV.B says are taken from prior work based on code-davinci-002. Section V.A(1) then treats these as comparable baselines, but the comparison is confounded by both model generation and prompt protocol; the statement that the authors 'uniformly select' the newer models to eliminate foundational-model discrepancies applies only to their own runs, not to the borrowed rows.
  4. [Section IV.C, Eq. (16)] The SC-accuracy metric as defined, N_selected-correct / N_total-correct, conditions on the set of programs that were already repaired successfully, so it does not measure the probability that the most frequent patch is correct across the full dataset. Consequently, the B2Fm result in Table II (T3 at 44.68% versus CoT at 54.64%) is not a direct measure of the reliability of T3's voting mechanism, and the RQ2 discussion should either redefine the metric over all programs or clearly label it as a conditional measure.
  5. [Section V.A(3), Table I] The text says 'GPT-3.5-turbo performs slightly better than GPT-4o-mini (46.70% vs 48.20%)' on B2Fs, but 48.20 is larger than 46.70, so the sentence contradicts Table I. The following claim that the pattern reverses on B2Fm is also wrong: gpt-4o-mini is again better (32.10% versus 28.20%). This error undermines the paragraph's argument about model-architecture adaptability and should be corrected.
  6. [Section V.C, Table III] The ablation conditions 'w/o plan' and 'w/o cause' are not defined: the text does not state which stage is removed, whether the forest size and top-n selection are held fixed, or how the removed stage's outputs are replaced. As reported, the ablation does not isolate the contribution of planning or causal analysis, and the conclusions in RQ3 go beyond what Table III can support.
minor comments (6)
  1. [Section II.C, Ref. [12]] The Tree-of-Thought baseline is cited to a blog post by D. Hulbert (May 2023) rather than to the original Tree-of-Thought paper (Yao et al., 2023); please cite the primary source.
  2. [Section V.B, Table II] The entry '43.7555.16' is missing a space or delimiter; it should read '43.75 55.16'.
  3. [Section V.D] The phrase 'these findings' should be capitalized as 'These findings'.
  4. [Sections III.B and III.C, Eqs. (4) and (10)] The notation T_i = {CoT(...)}_{j=1}^{N_i} is ambiguous; please clarify whether these are independent samples from the same prompt or distinct reasoning paths.
  5. [Section V.A(4)] The claimed 'average improvement of 14.90%' is not derivable from Table I as presented; specify the baseline set and averaging method.
  6. [Section V.D, Fig. 3] The figure as included has no axis labels or legend, and the shot-count protocol (number of demonstrations, which demonstrations) is not described; the plateau claims in RQ4 cannot be evaluated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central repair-rate comparison is an empirical measurement against an external correctness signal, with no fitted parameter or self-citation chain forcing the reported gains.

full rationale

The paper's central claim is a head-to-head empirical comparison on MODIT using a fixed pipeline (BM25 retrieval, multi-tree cause analysis, multi-tree repair planning, and patch generation) with no trainable parameters fitted to the test data. The repair rate R = N_correct/N_total in Eq. (15) depends on an external correctness oracle, so even though the paper does not specify the oracle or the rule that maps 30 generated samples to a single repaired/not-repaired label, that omission is a reproducibility and correctness risk, not circularity. The baseline numbers are taken from independent sources: S-C and S-C+BM25 from reference [31], NatGen from reference [33], and CoT/ToT/Plan-and-Solve/Analogical from the cited prompting literature; reference [31] is not authored by the present paper's authors. The only reference containing two co-authors of this paper is [2], and it is used only as an example of LLM-related prior work, not as a load-bearing premise. Eq. (16) defines SC accuracy as the correctness of the most frequently generated patch divided by the number of successful repairs, and although T3 uses frequency to rank causes and plans in Eqs. (5) and (11), the final patch is produced from selected top-n causes/plans via Eq. (14), not by majority voting over final patches. Thus the SC metric does not coincide with T3's selection rule by construction. No equation in the paper reduces to another by definition, and no fitted parameter or self-citation is renamed as a prediction. The derivation chain is therefore self-contained with respect to circularity, and any concerns about unstated validation protocols belong under correctness/reproducibility rather than circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework's empirical claims rest on several unstated engineering choices: the number of reasoning trees, candidates per tree, retrieval count, and top-n selection. These are not fitted parameters in an optimization sense, but they are degrees of freedom chosen by the authors that materially affect reported repair rates. The paper also assumes that test-based correctness and majority voting are reliable signals, with the latter partially contradicted by its own SC accuracy results.

free parameters (5)
  • M (number of reasoning trees) = not reported
    Equations (3) and (9) construct forests of M trees; M is never stated in Section III or IV.
  • N_i (candidates per tree) = not reported
    The number of error causes and repair plans generated per tree is undefined, and it directly affects diversity and voting outcomes.
  • k (BM25 retrieval count) = not reported
    Equation (1) retrieves k examples; the main experiments do not report k, and only RQ4 varies shot count.
  • top-n selected causes and plans = not reported
    Equations (7) and (13) select top-n items, but no numeric values are given anywhere in the paper.
  • temperature = 0.7
    Sampling temperature is fixed at 0.7, chosen by hand, and it influences the variance of generated patches.
assumptions (4)
  • domain assumption MODIT bug datasets have an oracle that determines correct repairs
    Repair rate in Eq. (15) requires a correctness oracle, but Section IV.A and IV.C never describe the test harness, so the reliability of the oracle is unverified.
  • domain assumption Majority frequency across reasoning paths approximates correctness
    Equations (5)-(8) and (11)-(13) select causes and plans by co-occurrence frequency, assuming shared outputs are more likely correct. Table II shows this prior is violated on B2Fm for T3, where SC accuracy is below CoT.
  • domain assumption BM25 lexical similarity retrieves relevant repair examples
    Equation (1) assumes text-matching code snippets provide useful context for repair, but no retrieval quality analysis is given.
  • domain assumption LLM outputs at temperature 0.7 with 30 samples are representative
    No repeated runs or confidence intervals are reported, so the stability of the point estimates is unknown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $T^3$: Multi-level Tree-based Automatic Program Repair with Large Language Models." pith.science (2026). https://pith.science/paper/NBK3VTJI

@misc{pith2026250621211,
  author       = {Pith},
  title        = {Pith review of: $T^3$: Multi-level Tree-based Automatic Program Repair with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBK3VTJI}},
  note         = {Machine review of arXiv:2506.21211}
}
abstract

Automatic Program Repair (APR) is a core technology in software development and maintenance, with aims to enable automated defect repair with minimal human intervention. In recent years, the substantial advancements in Large Language Models (LLMs) and the Chain-of-Thought (CoT) techniques have significantly enhanced the reasoning capabilities of these models. However, due to the complex logic and multi-step reasoning ability needed, the application of CoT techniques in the APR domain remains insufficient. This study systematically evaluates the performance of several common CoT techniques in APR tasks and proposes an innovative framework $T^3$, which integrates the powerful reasoning capabilities of LLMs with tree search, effectively improving the precision of generating candidate repair solutions. Furthermore, $T^3$ provides valuable guidance for optimizing sample selection and repair strategies in APR tasks, establishing a robust framework for achieving efficient automated debugging.

Figures

Figures reproduced from arXiv: 2506.21211 by the authors.

Figure 1
Figure 1. The general process of Automated Program Repair includes the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The T 3 method involves the following structured steps: 1.Sample Retrieval: Perform a keyword search utilizing the BM25 algorithm to identify relevant code examples from the codebase that are similar to the target buggy snippet. 2.Cause Analysis: Analyze the identified examples alongside the target buggy snippet to determine the underlying causes of the error. 3.Repair Plan Generation: Formulate a detailed repair st… view at source ↗
Figure 3
Figure 3. Repair Accuracy Performance under Different Shot Quantities [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 22 canonical work pages

  1. [1]

    A systematic literature review on large language models for automated program repair,

    Q. Zhang, C. Fang, Y . Xie, Y . Ma, W. Sun, Y . Yang, and Z. Chen, “A systematic literature review on large language models for automated program repair,”arXiv preprint arXiv:2405.01466, 2024

    arXiv 2024

  2. [2]

    Atomic fact decomposition helps attributed question answering,

    Z. Yan, J. Wang, J. Chen, X. Li, R. Li, and J. Z. Pan, “Atomic fact decomposition helps attributed question answering,”arXiv preprint arXiv:2410.16708, 2024

    arXiv 2024

  3. [3]

    Nmt vs mlm: Which is the best paradigm for apr?

    C. YiHeng, K. Huang, H. Wanget al., “Nmt vs mlm: Which is the best paradigm for apr?” in2024 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2024, pp. 1–7

    work page 2024

  4. [4]

    Rap-gen: Retrieval- augmented patch generation with codet5 for automatic program repair,

    W. Wang, Y . Wang, S. Joty, and S. C. Hoi, “Rap-gen: Retrieval- augmented patch generation with codet5 for automatic program repair,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2023, pp. 146–158

    work page 2023

  5. [5]

    Applying codebert for automated pro- gram repair of java simple bugs,

    E. Mashhadi and H. Hemmati, “Applying codebert for automated pro- gram repair of java simple bugs,” in2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 505–509

    work page 2021

  6. [6]

    CodeT5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “CodeT5+: Open code large language models for code understanding and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 1069–1088

    work page 2023

  7. [7]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8696–8708

    work page 2021

  8. [8]

    CodeBERT: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” inFindings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y . He, and Y . Liu, Eds., 2020, pp. 1536–1547

    work page 2020

Show all 33 references
  1. [9]

    Less training, more repairing please: revisiting automated program repair via zero-shot learning,

    C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2022, pp. 959– 971

    2022

  2. [10]

    Gamma: Revisiting template-based automated program repair via mask predic- tion,

    Q. Zhang, C. Fang, T. Zhang, B. Yu, W. Sun, and Z. Chen, “Gamma: Revisiting template-based automated program repair via mask predic- tion,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 535–547

    2023

  3. [11]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for 0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for 0.42 each using chatgpt,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 819–831

    2024

  4. [12]

    Using tree-of-thought prompting to boost chatgpt’s reason- ing,

    D. Hulbert, “Using tree-of-thought prompting to boost chatgpt’s reason- ing,” May 2023

    2023

  5. [13]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long...

    2023

  6. [14]

    Large language models as analogical reasoners,

    M. Yasunaga, X. Chen, Y . Li, P. Pasupat, J. Leskovec, P. Liang, E. H. Chi, and D. Zhou, “Large language models as analogical reasoners,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  7. [15]

    Search-based efficient automated program repair using mutation and fault localization,

    S. Sun, J. Guo, R. Zhao, and Z. Li, “Search-based efficient automated program repair using mutation and fault localization,” in2018 IEEE 42nd Annual Computer Software and Applications Conference (COMP- SAC), vol. 1. IEEE, 2018, pp. 174–183

    2018

  8. [16]

    Speeding up constraint-based program repair using a search-based technique,

    J. Yi and E. Ismayilzada, “Speeding up constraint-based program repair using a search-based technique,”Information and Software Technology, vol. 146, p. 106865, 2022

    2022

  9. [17]

    Tbar: Revisiting template-based automated program repair,

    K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyand ´e, “Tbar: Revisiting template-based automated program repair,” inProceedings of the 28th ACM SIGSOFT international symposium on software testing and anal- ysis, 2019, pp. 31–42

    2019

  10. [18]

    Dear: A novel deep learning-based approach for automated program repair,

    Y . Li, S. Wang, and T. N. Nguyen, “Dear: A novel deep learning-based approach for automated program repair,” inProceedings of the 44th international conference on software engineering, 2022, pp. 511–523

    2022

  11. [19]

    Conversational automated program repair,

    C. S. Xia and L. Zhang, “Conversational automated program repair,” arXiv preprint arXiv:2301.13246, 2023

    2023 arXiv

  12. [20]

    Large language models-guided dynamic adaptation for temporal knowledge graph reasoning,

    J. Wang, S. Kai, L. Luo, W. Wei, Y . Hu, A. W.-C. Liew, S. Pan, and B. Yin, “Large language models-guided dynamic adaptation for temporal knowledge graph reasoning,”Advances in Neural Information Processing Systems, vol. 37, pp. 8384–8410, 2024

    2024

  13. [21]

    Made: Multicurvature adaptive embedding for temporal knowledge graph completion,

    J. Wang, B. Wang, J. Gao, S. Pan, T. Liu, B. Yin, and W. Gao, “Made: Multicurvature adaptive embedding for temporal knowledge graph completion,”IEEE Transactions on Cybernetics, 2024

    2024

  14. [22]

    Ime: Integrating multi-curvature shared and specific embedding for temporal knowledge graph completion,

    J. Wang, Z. Cui, B. Wang, S. Pan, J. Gao, B. Yin, and W. Gao, “Ime: Integrating multi-curvature shared and specific embedding for temporal knowledge graph completion,” inProceedings of the ACM Web Conference 2024, 2024, pp. 1954–1962

    2024

  15. [23]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  16. [24]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,”arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  17. [25]

    A systematic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” inProceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10

    2022

  18. [26]

    Prompt programming for large language models: Beyond the few-shot paradigm,

    L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” inExtended abstracts of the 2021 CHI conference on human factors in computing systems, 2021, pp. 1–7

    2021

  19. [27]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,”arXiv preprint arXiv:2402.07927, 2024

    2024 arXiv

  20. [28]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  21. [29]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  22. [30]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Better patching using llm prompting, via self-consistency,

    T. Ahmed and P. Devanbu, “Better patching using llm prompting, via self-consistency,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1742–1746

    2023

  24. [32]

    An empirical study on learning bug-fixing patches in the wild via neural machine translation,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 4, pp. 1–29, 2019

    2019

  25. [33]

    Natgen: generative pre-training by “naturalizing

    S. Chakraborty, T. Ahmed, Y . Ding, P. T. Devanbu, and B. Ray, “Natgen: generative pre-training by “naturalizing” source code,” inProceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 18–30

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.