Pith. sign in

REVIEW 4 major objections 5 minor 90 references

Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No single prompt-engineering technique consistently outperforms others across software engineering tasks, according to a four-model, ten-task benchmark.

desk verdict A broad but statistically fragile benchmark of prompting techniques for SE; worth a serious referee if the authors add significance testing. read the letter →

arxiv 2506.05614 v1 pith:7KPAY4BH submitted 2025-06-05 cs.SE

E. G. Santana Jr , Gabriel Benjamin , Melissa Araujo , Harrison Santos , David Freitas , Eduardo Almeida , Paulo Anselmo da M. S. Neto , Jiawei Li
show 2 more authors
Jina Chun Iftekhar Ahmed
This is my paper
classification cs.SE
keywords promptengineeringlargelanguagemodelssoftwaretaskscodegenerationunderstandingselectionresourcecostempiricalbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that the effectiveness of prompt engineering techniques for software engineering tasks is task- and model-dependent: no single technique consistently wins across all ten tasks and four LLMs. It reports that ES-KNN, which selects similar in-context examples by nearest-neighbor search, is the most frequent aggregate winner, while USC (universal self-consistency) leads on code question answering and code generation, and several techniques perform below a simple baseline. The study also links prompt linguistic features, such as lexical diversity and token count, to performance, and uses contrastive explanations to identify structured guidance and in-context examples as the factors most often behind a technique's success. Resource accounting shows that top performers can be token- or time-heavy, so choosing a technique involves a cost-performance trade-off.

What carries the argument

The load-bearing instrument is a controlled comparison corpus: 14 prompting techniques drawn from six prompt-engineering dimensions, applied to 10 SE tasks sampled from a prior benchmark, with each technique instantiated by ten paraphrase templates produced by a commercial LLM and filtered by pairwise researcher review (Cohen's kappa = 0.45). Every technique-task pair is run on four LLMs, and performance per task is aggregated across models and z-scored to compare metrics of different scales. Linguistic metrics (lexical diversity, token count, readability indices) are correlated with performance via Spearman correlation, and contrastive explanations, where the LLM is asked why the best technique beats the worst, are hand-coded into factor categories such as Structured Guidance and In-Context Examples. Token counts and response times are collected per prompt to rank resource efficiency.

What would settle it

Run the identical benchmark with a fresh set of paraphrases per technique produced independently and validated with inter-rater agreement above 0.8; if one technique then wins all ten tasks across the four models, the paper's Observation 1 is overturned. If the rankings instead reshuffle, the no-universal-winner result survives but technique-level attributions would need to be treated with caution.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is Observation 1: no prompting technique consistently outperforms all others across SE tasks, and some techniques can hurt performance. Aggregating four LLMs per task, ES-KNN is best for Clone Detection, Code Translation, and Assert Generation; USC is best for Code QA and Code Generation; ToT (Tree of Thought) is best for Defect Detection; and for some tasks the winning technique is the control baseline itself. The worst techniques often score below the simple baseline, and the best techniques differ for the o3-mini model, with Role Prompting winning most tasks there. Linguistic analysis finds lexical diversity positively correlated with performance and token count negatively correlated, with readability correlations reversing between code understanding and code generation tasks.

Load-bearing premise

The rankings hold only if the ten paraphrases per technique, filtered with researcher agreement of Cohen's kappa 0.45, are faithful instances of the technique rather than incidental wording differences.

Editorial extensions

If this is right

  • Practitioners should select prompting techniques per task and per model rather than adopting a single prompt recipe, since no technique won across all ten tasks.
  • ES-KNN is a strong default for clone detection, code translation, and assert generation, while USC is a strong default for code question answering and code generation.
  • Some published prompting techniques can underperform a simple baseline, so prompt engineering is not automatically beneficial and should be validated before adoption.
  • Resource-aware selection matters: ES-KNN is fast but token-hungry, USC and Self-Refine add latency, and Role Prompting is the most token-efficient option.
  • The winning techniques shift for o3-mini, suggesting that prompt effectiveness does not transfer across model scales or families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the ten paraphrases per technique were filtered with only moderate inter-rater agreement (kappa = 0.45), the observed rankings may partly reflect wording artifacts; a replication with stricter validation could shrink or shift the gaps between techniques.
  • Editorial inference: the linguistic correlations suggest a testable design rule—reduce token count while increasing lexical diversity—that could be applied to prompts outside the 14 techniques studied.
  • Editorial inference: the task-model specificity of the winners implies that an automated prompt selector trained on task embeddings and model identity could outperform any fixed technique; the released dataset would support such a meta-model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports an empirical comparison of 14 prompting techniques applied to 10 software-engineering tasks using four LLMs (DeepSeek-V3, Qwen2.5-Coder-32B-Instruct, Llama-3.3-70B-Instruct, and OpenAI o3-mini). For each technique-task-model cell the authors run a single pass over a sampled dataset, compute task-specific metrics (accuracy, F1, BLEU, CodeBLEU), and rank the techniques. They also compute correlations between five linguistic prompt features and performance (RQ2), solicit contrastive explanations from the same LLMs to explain why the best technique outperformed the worst (RQ3), and report token and time savings (RQ4). The headline finding is Observation 1: no single prompting technique consistently outperforms all others, and some techniques perform below a baseline; ES-KNN is the most frequent aggregate winner, while USC is best for Code QA and Code Generation. The authors release their data and scripts.

Significance. If its central claims held, the paper would give practitioners a useful evidence-based map of prompt-technique performance across common SE tasks and models, and the cost analysis would address a real operational concern. The study is comparatively broad (14 techniques × 10 tasks × 4 models) and the public artifact release is a strength. However, the current statistical reporting is not sufficient to support the ranking-based conclusions: single runs without variance information cannot establish which differences are real, and the RQ2/RQ4 analyses omit key definitions and independence assumptions. The empirical core is plausible but needs substantial strengthening before the practical recommendations can be accepted.

major comments (4)
  1. [Section IV-A / Section III-E] The central ranking results and Observation 1 rest on a single run per technique-task-model cell, with no confidence intervals, effect sizes, or significance tests. In Code QA, the top technique USC scores 55.67 versus control 50.99 on roughly 382 instances; the approximate standard error of a proportion near 0.55 is about 2.5 percentage points, so the 4.68-point gap is on the order of 1.3 standard errors and cannot be distinguished from sampling noise. In Code Summarization, the reported best technique SG-ICL (BLEU 4.15) is actually below the control (4.16), so the 'best' label is not even directionally consistent with the baseline. The z-scores in Table IV are descriptive standardizations, not inferential statistics. I request multi-run or bootstrap/permutation analyses and reporting of variance to support Observation 1, the worst-versus-baseline claims, and the task/model-specific recommendations.
  2. [Section IV-D / Tables V–VI] The 'tokens saved' and 'time saved' metrics are never defined. The text in Section IV-D explains that efficiency is computed by dividing performance by tokens and response time, but Tables V and VI report mean numbers of tokens and seconds 'saved per prompt' without specifying the reference (presumably the baseline prompt) or the unit over which the mean is taken. The reported value of 19,485 tokens saved for Code QA with Llama is not interpretable without this definition, and no variance or sample size is reported. This undermines RQ4 and Observation 4.
  3. [Section III-F / Section IV-B] The Spearman correlations in RQ2 do not state their unit of analysis. Section III-F says linguistic metrics are averaged per prompting technique for each SE task before correlating with performance, but the N used for the p-values is not reported. If the unit is technique-task combinations, the observations are not independent because each technique appears in every task and each task in every technique, so the reported p-values (e.g., r=0.4440, p<0.001) are unverifiable and likely anti-conservative. Please report N, degrees of freedom, and a multilevel analysis that accounts for technique and task as random effects.
  4. [Section III-G / Section IV-C] Observation 3 claims that prompting techniques show greater effectiveness when they include structured guidance and in-context examples, but this conclusion is based on contrastive explanations generated by the same LLMs that produced the task outputs, followed by manual categorization. The manuscript does not validate that these self-reports reflect actual causes; for instance, no intervention test adds or removes the alleged 'structured guidance' to confirm its effect. RQ3 is framed as 'according to LLMs,' and Observation 3 should be qualified accordingly or supported by a controlled prompt manipulation study.
minor comments (5)
  1. [Section III-B, item 1] The sentence describing embedding-model selection is incomplete: 'We selected the most freqency used and best performing' ends without an object or period.
  2. [Tables II and III] The 'Understanding Tasks' and 'Generation Tasks' headers appear merged with the first task rows in the rendered tables, making it difficult to associate each technique with the correct task.
  3. [Table IV] The table columns are not aligned with the header: for example, the Code Summarization row appears to list SG-ICL 4.15 versus control 4.16, which is inconsistent with Table II, where Control is the aggregate best for that task. Please reconcile the tables and clarify what the z-scores and raw values represent.
  4. [Section IV-D] 'Response time' is not defined: it is unclear whether this is end-to-end API latency, generation time only, or something else, and how network variability was handled.
  5. [Abstract / Section III-C] The claimed 'more than 2k prompts' is ambiguous: 14 techniques × 10 variations × 10 tasks × 4 models gives 5,600 prompt-template instances, while 14 × 10 × 4 = 560 prompt runs. Please clarify what counts as a 'prompt' in this number.

Circularity Check

1 steps flagged · score 4.0 of 10

Core benchmark rankings are independent, but the RQ3 contrastive-explanation factors restate the definitions of the winning techniques.

  1. self definitional [Section III-G (Contrastive Explanation) and Section IV-C (RQ3), Observation 3]
    "The findings from the contrastive explanation analysis suggest that Structured Guidance and In-Context Examples are the most prevalent factors among the best-performing techniques."

    The best-performing techniques include ES-KNN, which Section III-B.1 defines as 'Selects exemplars using a k-nearest neighbor approach to enrich the prompt context.' Section III-G states that for each task 'we included the definitions of the prompting technique that performed the best and the worst' when prompting the LLM, and only showed examples where the superior technique was already significantly better. Asking the same LLM to compare a technique whose defining feature is in-context examples against one without them, and then coding the answer as 'In-Context Example,' is a restatement of the technique's definition rather than an independent discovery. 'Structured Guidance' is likewise a generic label for any instruction-following prompt.

full rationale

The paper's central empirical contribution—the RQ1 comparison of 14 prompting techniques over 10 SE tasks and 4 LLMs against a baseline—is self-contained: it uses external benchmarks, no fitted parameters, and no load-bearing self-citations. The authors' self-citations (e.g., references [4], [5], [20]) are related-work context, not used to justify the ranking. RQ2 correlations and RQ4 cost measurements are also direct measurements, not predictions from fitted quantities. The only identifiable circularity is in RQ3: the 'factors' (Structured Guidance, In-Context Example) are obtained by asking the model to contrast the best and worst techniques while explicitly supplying their definitions, and the dominant factor for ES-KNN is literally the mechanism that defines ES-KNN. This makes Observation 3 partially definitional, but it does not undermine Observations 1, 2, or 4. The lack of significance testing and single-run design noted in the threats section is a correctness/robustness concern, not a circularity concern. Overall score 4 reflects a secondary, partial definitional circularity in the explanation analysis while the main benchmark findings remain independent.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The empirical core introduces no fitted parameters and no invented entities. The main subjective choices are the hand-picked model temperatures, the unreported k in ES-KNN, and several behavioral assumptions about sampling, single-run stability, readability metrics, and LLM self-explanations. None of these is parameter-fitted to make a result appear, but the first two are necessary for reproducing the reported rankings.

free parameters (2)
  • k in ES-KNN
    The top-k nearest-neighbor exemplar count for ES-KNN is never specified; the paper only says 'retrieve the top-k nearest neighbor samples' (Section III-B). Without k, the method cannot be exactly reproduced and k may materially affect exemplar content and performance.
  • Softmax temperature = 1 for o3-mini and DeepSeek-V3; 0.7 for Llama-3.3-70B and Qwen2.5-Coder-32B
    Temperature is fixed by hand and differs across models (Section VI, Internal Validity). This is a methodological choice, not fitted to task data, but it influences stochasticity and comparability of outputs across models.
assumptions (5)
  • domain assumption The random samples per dataset, sized for a 95% confidence level and 5% margin of error, are representative and stable enough for ranking prompting techniques.
    Section III-A describes random sampling with a 95% confidence level and 5% margin of error; the paper does not test whether the sampled subsets yield stable rankings under resampling.
  • domain assumption A single generation per instance at fixed temperature is sufficient; output stochasticity does not change the rankings.
    Section III-E and Section VI state fixed temperatures, but no repeated runs or variance estimates are reported, so ranking stability is assumed.
  • ad hoc to paper LLM-generated contrastive explanations reflect the actual causes of performance differences between prompting techniques.
    Section III-G collects explanations by prompting the same LLMs with in-context examples; Section IV-C converts these self-reports into causal-sounding factors. No external validation is provided.
  • domain assumption Human readability metrics (Flesch-Kincaid, Gunning Fog, Flesch Reading Ease) are meaningful predictors of LLM performance.
    Section III-F applies readability indices designed for human readers to prompts for LLMs; the paper itself notes correlations differ by task and model, but the assumption that these metrics capture prompt complexity relevant to LLMs is untested.
  • domain assumption BLEU and CodeBLEU are valid automatic metrics for the generation tasks and correlate with actual correctness.
    Section III-A uses CodeBLEU for translation, bug fixing, and code generation, and BLEU for mutant, assert, and summarization tasks; these are standard in the field but known to be imperfect proxies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks." pith.science (2026). https://pith.science/paper/7KPAY4BH

@misc{pith2026250605614,
  author       = {Pith},
  title        = {Pith review of: Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KPAY4BH}},
  note         = {Machine review of arXiv:2506.05614}
}
read the original abstract

A growing variety of prompt engineering techniques has been proposed for Large Language Models (LLMs), yet systematic evaluation of each technique on individual software engineering (SE) tasks remains underexplored. In this study, we present a systematic evaluation of 14 established prompt techniques across 10 SE tasks using four LLM models. As identified in the prior literature, the selected prompting techniques span six core dimensions (Zero-Shot, Few-Shot, Thought Generation, Ensembling, Self-Criticism, and Decomposition). They are evaluated on tasks such as code generation, bug fixing, and code-oriented question answering, to name a few. Our results show which prompting techniques are most effective for SE tasks requiring complex logic and intensive reasoning versus those that rely more on contextual understanding and example-driven scenarios. We also analyze correlations between the linguistic characteristics of prompts and the factors that contribute to the effectiveness of prompting techniques in enhancing performance on SE tasks. Additionally, we report the time and token consumption for each prompting technique when applied to a specific task and model, offering guidance for practitioners in selecting the optimal prompting technique for their use cases.

Figures

Figures reproduced from arXiv: 2506.05614 by the authors.

Figure 1
Figure 1. Example of two variations of the emotion prompting template for the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 46 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani and et al., “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  2. [2]

    Lost in translation: A study of bugs introduced by large language models while translating code,

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in translation: A study of bugs introduced by large language models while translating code,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  3. [3]

    Transagents: Build your translation company with language agents,

    M. Wu, J. Xu, and L. Wang, “Transagents: Build your translation company with language agents,” in Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, 2024, pp. 131–141

    2024

  4. [4]

    Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,

    J. Li, D. Farag ´o, C. Petrov, and I. Ahmed, “Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 745–766, 2024

    2024

  5. [5]

    Consider what humans consider: Optimizing commit mes- sage leveraging contexts considered by human,

    ——, “Consider what humans consider: Optimizing commit mes- sage leveraging contexts considered by human,” arXiv preprint arXiv:2503.11960, 2025

    arXiv 2025

  6. [6]

    Towards an understanding of large language models in software engineering tasks,

    S. Hou, Y . Liu, and J. Lee, “Towards an understanding of large language models in software engineering tasks,” in 42nd International Conference on Software Engineering (ICSE) , 2024, pp. 754–768

    2024

  7. [7]

    Aligning the objective of llm-based program repair,

    J. Xu, Y . Fu, S. H. Tan, and P. He, “Aligning the objective of llm-based program repair,” arXiv preprint arXiv:2404.08877 , 2024

    arXiv 2024

  8. [8]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,” in International Conference on Learning Representations (ICLR) , 2024

    2024

Show all 90 references
  1. [9]

    The prompt report: A systematic survey of prompt engineering techniques,

    S. S. et al., “The prompt report: A systematic survey of prompt engineering techniques,” 2025. [Online]. Available: https://arxiv.org/ abs/2406.06608

    2025 arXiv

  2. [10]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Comput. Surv. , vol. 55, no. 9, Jan

  3. [11]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22...

    2022

  4. [12]

    An empirical comparison of pre-trained models of source code,

    C. Niu, C. Li, V . Ng, D. Chen, J. Ge, and B. Luo, “An empirical comparison of pre-trained models of source code,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2136–2148

    2023

  5. [13]

    Contrastive explanations for model interpretability,

    A. Jacovi, S. Swayamdipta, S. Ravfogel, Y . Elazar, Y . Choi, and Y . Goldberg, “Contrastive explanations for model interpretability,”arXiv preprint arXiv:2103.01378, 2021

    2021 arXiv

  6. [14]

    Llm in a flash: Efficient large language model inference with limited memory,

    K. Alizadeh, S. I. Mirzadeh, D. Belenko, S. Khatamifard, M. Cho, C. C. Del Mundo, M. Rastegari, and M. Farajtabar, “Llm in a flash: Efficient large language model inference with limited memory,” pp. 12 562–12 584, 2024

    2024

  7. [15]

    Exploring the potential of chatgpt in automated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Exploring the potential of chatgpt in automated code refinement: An empirical study,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–13

    2024

  8. [16]

    Harnessing large language models for curated code reviews,

    O. B. Sghaier, M. Weyssow, and H. Sahraoui, “Harnessing large language models for curated code reviews,” arXiv preprint arXiv:2502.03425, 2025

    2025 arXiv

  9. [17]

    Ai that builds with you,

    G. Copilot, “Ai that builds with you,” 2023. [Online]. Available: https://github.com/features/copilot

    2023

  10. [18]

    Jetbrains ai: Optimize your workflow. with ai built for you,

    J. AI, “Jetbrains ai: Optimize your workflow. with ai built for you,”

  11. [19]

    Jtype less, code more. visual studio intellicode brings ai assistance directly into your personal development flow

    V . S. IntelliCode, “Jtype less, code more. visual studio intellicode brings ai assistance directly into your personal development flow.”

  12. [20]

    Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,

    A. Sergeyuk, Y . Golubev, T. Bryksin, and I. Ahmed, “Using ai-based coding assistants in practice: State of affairs, perceptions, and ways forward,” Information and Software Technology , vol. 178, p. 107610, 2025

    2025

  13. [21]

    Large language model fine-tuning with low-rank adaptation: A performance exploration,

    B. Hanindhito, B. Patel, and L. K. John, “Large language model fine-tuning with low-rank adaptation: A performance exploration,” in Proceedings of the 16th ACM/SPEC International Conference on Performance Engineering (ICPE ’25) . Toronto, ON, Canada: ACM, 2025, pp. 92–104. [On...

    2025 doi

  14. [22]

    Available: https://visualstudio.microsoft.com/services/ intellicode/

    [Online]. Available: https://visualstudio.microsoft.com/services/ intellicode/

  15. [23]

    A large-scale empirical study on fine-tuning large language models for unit testing,

    Y . Shang, Q. Zhang, C. Fang, S. Gu, J. Zhou, and Z. Chen, “A large-scale empirical study on fine-tuning large language models for unit testing,” 2024. [Online]. Available: https://arxiv.org/abs/2412.16620

    2024 arXiv

  16. [24]

    Prompting or fine-tuning? a comparative study of large language models for taxonomy construction,

    B. Chen, F. Yi, and D. Varr ´o, “Prompting or fine-tuning? a comparative study of large language models for taxonomy construction,” in 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) . IEEE, 2023, pp. 588–596

    2023

  17. [25]

    Language models are few-shot learners,

    T. B. e. a. Brown, “Language models are few-shot learners,” in Pro- ceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Curran Associates Inc., 2020

    2020

  18. [26]

    Language models in software development tasks: An experimental analysis of energy and accuracy,

    N. Alizadeh, B. Belchev, N. Saurabh, P. Kelbert, and F. Castor, “Language models in software development tasks: An experimental analysis of energy and accuracy,” 2025. [Online]. Available: https: //arxiv.org/abs/2412.00329

    2025 arXiv

  19. [27]

    Prompting is programming: A query language for large language models,

    P. Hase, I. Purohit, and M. Bansal, “Prompting is programming: A query language for large language models,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2023. [Online]. Available: https://openreview.net/forum?id=RIu5lyNXjT

    2023

  20. [28]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” arXiv preprint arXiv:2406.00515, 2024

    2024 arXiv

  21. [29]

    Interpreting language models with contrastive explanations,

    K. Yin and G. Neubig, “Interpreting language models with contrastive explanations,” arXiv preprint arXiv:2202.10419 , 2022

    2022 arXiv

  22. [30]

    Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval,

    Y . Liu, R. Meng, S. Joty, S. Savarese, C. Xiong, Y . Zhou, and S. Yavuz, “Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval,” arXiv preprint arXiv:2411.12644 , 2024

    2024 arXiv

  23. [31]

    Distinguish before answer: Generating contrastive explanation as knowledge for commonsense question answering,

    Q. Chen, G. Xu, M. Yan, J. Zhang, F. Huang, L. Si, and Y . Zhang, “Distinguish before answer: Generating contrastive explanation as knowledge for commonsense question answering,” arXiv preprint arXiv:2305.08135, 2023

    2023 arXiv

  24. [32]

    Learning and evaluating contextual embedding of source code,

    A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in Proceedings of the 37th International Conference on Machine Learning , ser. ICML’20. JMLR.org, 2020

    2020

  25. [33]

    Towards a big data curated benchmark of inter-project code clones,

    J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, “Towards a big data curated benchmark of inter-project code clones,” in 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 476–480

    2014

  26. [34]

    voyage-code-2: Elevate your code retrieval,

    V . AI, “voyage-code-2: Elevate your code retrieval,”

  27. [35]

    Available: https://blog.voyageai.com/2024/01/23/ voyage-code-2-elevate-your-code-retrieval/

    [Online]. Available: https://blog.voyageai.com/2024/01/23/ voyage-code-2-elevate-your-code-retrieval/

    2024

  28. [36]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2009.10297

    2020 arXiv

  29. [37]

    An empirical study on learning bug-fixing patches in the wild via neural machine translation,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 4, pp. 1–29, 2019

    2019

  30. [38]

    CoSQA: 20,000+ web queries for code search and question answering,

    J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, and N. Duan, “CoSQA: 20,000+ web queries for code search and question answering,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021

  31. [39]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. L. et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” 2021. [Online]. Available: https://arxiv.org/abs/2102.04664

    2021 arXiv

  32. [40]

    On learning meaningful assert statements for unit test cases,

    C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , ser. ICSE ’20. New York, NY , USA: Association for Computing Mac...

    2020

  33. [41]

    Mapping language to code in programmatic context,

    S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Mapping language to code in programmatic context,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Asso...

    2018

  34. [42]

    A generative and mutational approach for synthesizing bug- exposing test cases to guide compiler fuzzing,

    G. Ye, T. Hu, Z. Tang, Z. Fan, S. H. Tan, B. Zhang, W. Qian, and Z. Wang, “A generative and mutational approach for synthesizing bug- exposing test cases to guide compiler fuzzing,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on t...

    2023

  35. [43]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W. jing Zhu, “Bleu: a method for automatic evaluation of machine translation,” 2002, pp. 311–318

    2002

  36. [44]

    Selective annotation makes language models better few-shot learners,

    H. Su, J. Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, R. Zhang, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu, “Selective annotation makes language models better few-shot learners,” 2022. [Online]. Available: https://arxiv.org/abs/2209.01975

    2022 arXiv

  37. [45]

    Explor- ing demonstration ensembling for in-context learning,

    M. Khalifa, L. Logeswaran, M. Lee, H. Lee, and L. Wang, “Explor- ing demonstration ensembling for in-context learning,” arXiv preprint arXiv:2308.08780, 2023

    2023 arXiv

  38. [46]

    Qualitative content analysis,

    J. Forman and L. Damschroder, “Qualitative content analysis,” in Em- pirical methods for bioethics: A primer . Emerald Group Publishing Limited, 2007, pp. 39–62

    2007

  39. [47]

    What makes good in-context examples for GPT-3?

    J. Liu, D. Shen, Y . Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for GPT-3?” in Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures , E. Agirre, M. Apid...

    2022

  40. [48]

    Contrastive chain-of-thought prompting,

    Y . K. Chia, G. Chen, L. A. Tuan, S. Poria, and L. Bing, “Contrastive chain-of-thought prompting,” 2023. [Online]. Available: https://arxiv.org/abs/2311.09277

    2023 arXiv

  41. [49]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2305.10601

    2023 arXiv

  42. [50]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR) , 2023

    2023

  43. [51]

    jina-embeddings-v2-base-code,

    H. Face, “jina-embeddings-v2-base-code,” 2025. [Online]. Available: https://huggingface.co/jinaai/jina-embeddings-v2-base-code

    2025

  44. [52]

    Self-refine: Iterative refinement with self-feedback,

    A. M. et al., “Self-refine: Iterative refinement with self-feedback,”

  45. [53]

    Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator,

    H. J. Kim, H. Cho, J. Kim, T. Kim, K. M. Yoo, and S. goo Lee, “Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator,” 2022. [Online]. Available: https://arxiv.org/abs/2206.08082

    2022 arXiv

  46. [54]

    Measuring and narrowing the compositionality gap in language models,

    O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis, “Measuring and narrowing the compositionality gap in language models,” 2023. [Online]. Available: https://arxiv.org/abs/2210.03350

    2023 arXiv

  47. [55]

    Universal self-consistency for large language model generation,

    X. Chen, R. Aksitov, U. Alon, J. Ren, K. Xiao, P. Yin, S. Prakash, C. Sutton, X. Wang, and D. Zhou, “Universal self-consistency for large language model generation,” 2023. [Online]. Available: https://arxiv.org/abs/2311.17311

    2023 arXiv

  48. [56]

    Large language models understand and can be enhanced by emotional stimuli,

    C. Li, J. Wang, Y . Zhang, K. Zhu, W. Hou, J. Lian, F. Luo, Q. Yang, and X. Xie, “Large language models understand and can be enhanced by emotional stimuli,” 2023. [Online]. Available: https://arxiv.org/abs/2307.11760

    2023 arXiv

  49. [57]

    Available: https://arxiv.org/abs/2303.17651

    [Online]. Available: https://arxiv.org/abs/2303.17651

    arXiv

  50. [58]

    Rephrase and respond: Let large language models ask better questions for themselves,

    Y . Deng, W. Zhang, Z. Chen, and Q. Gu, “Rephrase and respond: Let large language models ask better questions for themselves,” 2024. [Online]. Available: https://arxiv.org/abs/2311.04205

    2024 arXiv

  51. [59]

    Thread of thought unraveling chaotic contexts,

    Y . Zhou, X. Geng, T. Shen, C. Tao, G. Long, J.-G. Lou, and J. Shen, “Thread of thought unraveling chaotic contexts,” 2023. [Online]. Available: https://arxiv.org/abs/2311.08734

    2023 arXiv

  52. [60]

    Take a step back: Evoking reasoning via abstraction in large language models,

    H. S. Zheng, S. Mishra, X. Chen, H.-T. Cheng, E. H. Chi, Q. V . Le, and D. Zhou, “Take a step back: Evoking reasoning via abstraction in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.06117

    2024 arXiv

  53. [61]

    Prompt engineering: How prompt vocabulary af- fects domain knowledge,

    D. Schreiter, “Prompt engineering: How prompt vocabulary af- fects domain knowledge,” Ph.D. dissertation, Georg-August-Universit ¨at G¨ottingen, 2024

    2024

  54. [62]

    Bounding the capabilities of large language models in open text generation with prompt constraints,

    A. Lu, H. Zhang, Y . Zhang, X. Wang, and D. Yang, “Bounding the capabilities of large language models in open text generation with prompt constraints,” 2023. [Online]. Available: https://arxiv.org/abs/ 2302.09185

    2023 arXiv

  55. [63]

    Investigating the accuracy of chatgpt as a writing error correction tool,

    W. Alsaweed and S. Aljebreen, “Investigating the accuracy of chatgpt as a writing error correction tool,” International Journal of Computer- Assisted Language Learning and Teaching , vol. 14, pp. 1–18, 12 2024

    2024

  56. [64]

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models,

    Z. M. Wang and et al., “Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.00746

    2024 arXiv

  57. [65]

    Large language models as analogical reasoners,

    M. Yasunaga, X. Chen, Y . Li, P. Pasupat, J. Leskovec, P. Liang, E. H. Chi, and D. Zhou, “Large language models as analogical reasoners,”

  58. [66]

    Available: https://arxiv.org/abs/2310.01714

    [Online]. Available: https://arxiv.org/abs/2310.01714

    arXiv

  59. [67]

    Qwen2.5-coder technical report,

    B. H. et al., “Qwen2.5-coder technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2409.12186

    2024 arXiv

  60. [68]

    Chatgpt: Optimizing language models for dialogue,

    OpenAI, “Chatgpt: Optimizing language models for dialogue,” 2023. [Online]. Available: https://openai.com/blog/chatgpt

    2023

  61. [69]

    Openai o3-mini system card,

    OpenAI, “Openai o3-mini system card,” https://openai.com/index/ o3-mini-system-card/, 2025, accessed: 2025-05-30

    2025

  62. [70]

    Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation,

    T. Fang, S. Yang, K. Lan, D. F. Wong, J. Hu, L. S. Chao, and Y . Zhang, “Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation,” 2023. [Online]. Available: https://arxiv.org/abs/2304.01746

    2023 arXiv

  63. [71]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” in Proceedings of the 37th International Conference on Neural Information Processing Systems , ser. NIPS ’23. Red Ho...

    2023

  64. [72]

    Deepseek-v3 technical report,

    A. Liu and et al., “Deepseek-v3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2412.19437

    2025 arXiv

  65. [73]

    The technique of clear writing,

    R. Gunning, “The technique of clear writing,” (No Title), 1952

    1952

  66. [74]

    Llama 3.3 70b instruct model card,

    Meta AI, “Llama 3.3 70b instruct model card,” https://huggingface.co/ meta-llama/Llama-3.3-70B-Instruct, 2024, accessed: 2025-05-30

    2024

  67. [75]

    Quality control in software documentation: Measurement of text comprehensibility,

    F. Lehner, “Quality control in software documentation: Measurement of text comprehensibility,” Information & Management , vol. 25, no. 3, pp. 133–146, 1993. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/037872069390036S

    1993

  68. [76]

    Together models,

    together.ai, “Together models,” 2025. [Online]. Available: https: //www.together.ai/models

    2025

  69. [77]

    Cutting the gordian knot: The moving-average type–token ratio (mattr),

    M. A. Covington and J. D. McFall, “Cutting the gordian knot: The moving-average type–token ratio (mattr),” Journal of Quantitative Lin- guistics, vol. 17, no. 2, pp. 94–100, 2010

    2010

  70. [78]

    Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,

    J. P. Kincaid, R. P. Fishburne, R. L. Rogers, and B. S. Chissom, “Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel,” Naval Technical Training Command Millington TN Research Branch, Tech....

    1975

  71. [79]

    What makes good in-context demonstrations for code intelligence tasks with llms?

    S. Gao, X.-C. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What makes good in-context demonstrations for code intelligence tasks with llms?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 761–773

    2023

  72. [80]

    A new readability yardstick

    R. Flesch, “A new readability yardstick.” Journal of applied psychology, vol. 32, no. 3, p. 221, 1948

    1948

  73. [81]

    Card-sorting: From text to themes,

    T. Zimmermann, “Card-sorting: From text to themes,” in Perspectives on data science for software engineering . Elsevier, 2016, pp. 137–141

    2016

  74. [82]

    The proof and measurement of association between two things,

    C. Spearman, “The proof and measurement of association between two things,” The American Journal of Psychology, vol. 15, no. 1, pp. 72–101, 1904

    1904

  75. [83]

    Logic and causal attribution

    D. J. Hilton, “Logic and causal attribution.” in Part of this chapter is the text of a paper read at the symposium, ” Attitudes and Attribution: A Symposium in Honour of Jos Jaspars, ” convened at the Annual Conference of the British Psychological Society, Sheffield, England, ...

    1986

  76. [84]

    Towards faithfully interpretable nlp sys- tems: How should we define and evaluate faithfulness?

    A. Jacovi and Y . Goldberg, “Towards faithfully interpretable nlp sys- tems: How should we define and evaluate faithfulness?” arXiv preprint arXiv:2004.03685, 2020

    2004 arXiv

  77. [86]

    Advancing conversational ai: Best practices in prompt engineering for enhanced chatbot performance,

    S. Goriparthi, “Advancing conversational ai: Best practices in prompt engineering for enhanced chatbot performance,” Journal ID, vol. 6202, p. 8020

  78. [88]

    G. L. Iverson, Z Scores . New York, NY: Springer New York, 2011, pp. 2739–2740. [Online]. Available: https://doi.org/10.1007/ 978-0-387-79948-3 1263

    2011

  79. [89]

    Open coding descriptions,

    B. G. Glaser and Hon., “Open coding descriptions,” Grounded Theory Review: An International Journal , vol. 15, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:149583839

    2016

  80. [90]

    GitHub - prompt-study/prompt-tasks-study — github.com,

    “GitHub - prompt-study/prompt-tasks-study — github.com,” https:// github.com/prompt-study/prompt-tasks-study/tree/main, [Accessed 31- 05-2025]

    2025

  81. [2023]

    Available: https://doi.org/10.1145/3560815

    [Online]. Available: https://doi.org/10.1145/3560815

    doi

  82. [2024]

    Available: https://www.jetbrains.com/ai/

    [Online]. Available: https://www.jetbrains.com/ai/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.