Pith. sign in

REVIEW 5 major objections 5 minor 62 references

From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that prompting a large language model with lessons distilled from its own confident wrong answers lifts clone-detection F1 by up to 10.81 percentage points on a cross-language benchmark and 9.77 on a within-language…

desk verdict Real, partially out-of-sample F1 gains and a first error taxonomy for LLM clone detection, but no control prompt and an in-sample poolC half leave the causal story unproven. read the letter →

arxiv 2505.05679 v1 pith:UHAETMFM submitted 2025-05-08 cs.SE

QiHong Chen , Lianghao Jiang , Iftekhar Ahmed This is my paper · ORCID
classification cs.SE
keywords codeclonedetectionpromptbiaslargelanguagemodelsengineeringerroranalysissemanticclonescross-languagePaLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to show that a large language model's systematic mistakes on code-clone detection can be mined, classified, and converted into prompt instructions that fix a substantial share of those mistakes. On two clone-detection benchmarks, one within Python and one cross-language (Java/Python), the PaLM model scored F1 near 90 before any modification. The authors extracted the model's self-reported rationales for its confident wrong answers, grouped them into eight recurring error categories, and rewrote each category as a one-sentence 'lesson' appended to the prompt. Adding all eight lessons raised F1 by up to 10.81 points on the cross-language benchmark and 9.77 points on the within-language benchmark. If the result holds, it means a model's biases can be identified and partially corrected using nothing but its own explanations, with no retraining and no access to weights.

What carries the argument

The load-bearing object is the 'prompt bias lesson': a one-sentence instruction, derived from a mistake category, appended to the clone-detection prompt. The pipeline is: run the model with a default prompt, keep only confident wrong answers, ask the model to explain those errors in its own words, cluster the explanations into eight categories, translate each category into a lesson, and measure F1 before and after appending the lessons. Two human raters did the categorization and lesson-writing, with high reported inter-rater agreement. The clone-type taxonomy (identical, renamed, near-miss, semantic) supplies the interpretive frame: most categories describe the model treating semantic equivalence as non-clones because it overweights surface text.

What would settle it

Run the eight lessons against a control prompt of matched length and tone whose instructions are not derived from the model's errors, such as generic advice that code can be functionally equivalent despite textual differences: if the control prompt produces the same F1 improvement, the error-driven derivation is not what matters. Alternatively, build lessons from deliberately shuffled or fabricated rationales and check whether the improvement still appears.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt bias in LLM-based clone detection is not random fluctuation but a small set of recurring conceptual confusions, and that naming those confusions in the prompt recovers most of the lost performance. The eight categories range from a narrow reading of 'clone' (only identical code counts) to confusions about operators, variable naming, data structures, function and library names, thematic content, comprehension failures, and divergent implementation styles; most map to the model failing on Type 2-4 clones, especially semantic clones. The authors report that the most frequent category, misinterpretation of function and library API nomenclature, appears in 65.39% of wrong answers averaged across datasets. When all eight lessons were appended to the default prompt, PaLM's F1 rose from 89.30 to 96.48 on the avatar without-comment set and from 86.41 to 94.55 on poolC with-comment, with all gains reported as statistically significant; the single strongest lesson was that code logic outweighs minor code differences.

Load-bearing premise

The load-bearing premise is that PaLM's self-reported rationales for its wrong answers accurately reflect the true causes of those errors; if the rationales are post-hoc stories rather than genuine causes, the eight categories and lessons are not the reason the prompt helped.

Editorial extensions

If this is right

  • LLMs without fine-tuning can serve as strong clone detectors, reaching F1 scores near 90 on both within-language and cross-language benchmarks.
  • Prompt bias is classifiable: eight error categories account for the bulk of confident mistakes, and the same lessons generalize from the Python dataset to the Java/Python dataset.
  • The all-lessons prompt converts many more errors to correct answers than it introduces (for example 1,166 vs. 101 on poolC with comment), so the improvement is not a simple precision-recall trade.
  • Lesson 7 alone, which tells the model that code logic matters more than minor differences, gives the largest per-lesson gain on all four dataset variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A natural extension is to automate the loop, having the model propose lessons from its own rationales and selecting them by validation-set performance, which the paper flags as future work.
  • Inference: If the causal story is right, the same rationale-to-lesson procedure should transfer to other code tasks such as bug detection, code summarization, or security analysis, where models also show recurring, prompt-sensitive failure modes.
  • Inference: A placebo-prompt control (generic advice matched in length, not derived from observed errors) would separate the effect of domain knowledge injection from the effect of error-specific diagnosis; the paper does not report such a control.
  • Inference: The improvement may in part reflect that the lessons are effective generic reminders about clone semantics rather than corrections tied to the model's specific biases; comparing lessons derived from a different model's errors would test that.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies prompt bias of large language models (LLMs) for code clone detection. Using PaLM as the selected model, the authors run experiments on two datasets: poolC (within-language, Python clones) and an Avatar-derived cross-language dataset (Java/Python). From PaLM's incorrect predictions on poolC, they elicit high-level rationales, manually distill eight 'prompt bias mistake categories,' derive eight corresponding 'prompt bias lessons,' and append combinations of these lessons to the default prompt. They report F1 improvements up to 10.81% on Avatar and 9.77% on poolC and claim that the method is effective and statistically significant. The paper also reports prevalence statistics of the error categories and ablations of individual lessons.

Significance. If the central claim holds, the paper offers a practical and potentially generalizable recipe for improving LLM performance on clone detection by converting observed model errors into prompt instructions. The deterministic temperature setting, the explicit replication-package link, and the use of an out-of-sample Avatar evaluation for lessons derived from poolC are genuine strengths. The ablation study showing that some individual lessons hurt performance is also useful evidence of content specificity. However, the causal attribution of the F1 gains to the error-derived lessons is not established because of the missing control prompt and the in-sample nature of the poolC evaluation. These issues are fixable within the manuscript's scope, so the contribution is defensible but needs substantial revision.

major comments (5)
  1. [§3.5, Tables 7-8] The evaluation compares only the default prompt against prompts containing the error-derived lessons; there is no control prompt of matched length and style that conveys generic semantic-equivalence instructions without being derived from the model's errors. This matters because at the default operating point recall is the bottleneck: for example, avatar-with-comment recall rises from 76.75 to 97.64 and poolC-with-comment recall from 76.89 to 91.58 with all_lessons. Any instruction that pushes the model toward labeling more pairs as clones, such as 'textual differences should not determine clones,' would raise F1 even if the specific error taxonomy were irrelevant. The causal claim that the improved prompts work because they encode the eight identified error categories therefore requires a matched control prompt.
  2. [§3.2.3-§3.5, Tables 7-8] The poolC result is in-sample: the eight error categories and the lessons were derived from PaLM's errors on poolC, and the improved prompts were then evaluated on the same poolC test set. The reported 9.77% improvement for poolC is therefore not an out-of-sample estimate. The Avatar result is a genuine external check, but it only supports generalization to the cross-language setting. To validate the poolC claim, the authors should derive the categories and lessons on a training or validation subset and evaluate on a held-out test set, or at least report a cross-validation-style procedure.
  3. [§3.2.2, Table 3] The stated 80% confidence filter is inconsistent with the reported confidence distribution for poolC-with-comment. Table 3 shows that 100.00% of incorrect predictions in that dataset have confidence 0, so the 80%-confidence filter would retain zero poolC-with-comment errors. Yet Section 3.2.3 says the authors 'randomly selected 100 examples' from the reliable incorrect predictions to build the taxonomy. The provenance of those 100 examples is unclear: were they drawn only from poolC-without-comment, from a mix of datasets, or from unfiltered errors? This needs clarification because it affects the reproducibility of the taxonomy and the claimed relationship between the filter and the lessons.
  4. [§3.2.3, Figure 4] The entire taxonomy rests on the assumption that PaLM's self-reported high-level rationales for its wrong answers accurately reflect the true cause of the errors. LLMs often produce plausible post-hoc explanations that do not correspond to the actual computational cause of a misclassification. The paper does not validate the rationales against an independent, code-level analysis (for example, by checking whether the cited textual difference actually drove the prediction). If the rationales are not faithful, the eight categories and the lessons derived from them are not causally linked to the subsequent F1 improvements. The authors should either provide such validation, for instance by testing the lessons against rationales obtained from a different model or from controlled input perturbations, or explicitly weaken the causal claim.
  5. [§4.3, Tables 7-8] The statistical significance analysis uses a t-test for each lesson and each dataset individually, with many comparisons, but no multiple-comparison correction is reported. Given that eight lessons are tested on four datasets, some significant results would be expected by chance even if the lessons had no effect. Reporting adjusted p-values or a Bonferroni-style correction would strengthen the claim that the improvements are not artifacts of multiple testing.
minor comments (5)
  1. [Table 2] The CodeT5 accuracy for poolC with comment is reported as '0.0.45', which appears to be a typo and should read 0.45.
  2. [Table 4] The category 'Overemphasis On Textual Similarity' appears in the prevalence table but is not defined in the list of eight categories in Section 4.1.2; either add its definition or explain the discrepancy with the eight named categories.
  3. [§6] The threats-to-validity section acknowledges that the categories were derived from only 100 poolC instances and may not generalize, but it does not acknowledge the missing control prompt as a threat to the causal claim; this should be added and discussed explicitly.
  4. [§3.2.2] The rationale for choosing 80% as the confidence threshold is based on the observed distribution in Table 3, but that distribution is presented only later in Section 4.1.1; consider moving the table earlier or describing the distribution when the threshold is introduced.
  5. [§1] The claim that this is 'the first exploration of LLMs within the domain of clone detection' is too strong given the existing literature; the authors should temper this claim or provide a more careful review of prior LLM-based clone detection work.

Circularity Check

1 steps flagged · score 4.0 of 10

PoolC result is in-sample: lessons derived from poolC errors are evaluated on the same poolC test set; avatar provides an independent out-of-sample check.

  1. fitted input called prediction [Section 3.2.3 / Section 6 / Section 4.3 (Finding 3)]
    "To understand the root cause behind those rationales, we randomly selected 100 examples and reviewed them. ... In the process of defining prompt bias mistake categories, our approach mainly relied on the model's incorrect prediction groups within the poolC dataset, with a limited sample size of 100 instances. ... Notably, our analysis showed that the peak improvements of our methods can be up to 10.81% in terms of F1 score for the avatar dataset and 9.77% for the poolC dataset."

    The eight prompt-bias mistake categories were induced from PaLM's incorrect predictions on poolC (Section 3.2.3, 100 reviewed examples), and the prompt-bias lessons are explicitly a high-level encapsulation of the model's confusion (Section 3.4). The same poolC test set is then reused in Section 4.3 to report the 9.77% F1 improvement for poolC. Thus the poolC component of Finding 3 measures the method on the data that generated its input lessons; it is an in-sample evaluation, not a prediction. The avatar component is an external check because the categories were not derived from avatar errors, and the paper's own threats section says the categories relied on poolC.

full rationale

Score 4 rather than 6+ because the central claim is not entirely forced by the input. The avatar dataset is a genuine out-of-sample evaluation: lessons derived from poolC errors are applied to avatar and improve F1 by 10.81%, providing independent evidence that the error-derived lessons carry transferable content. The ablation (Tables 7-8) also shows individual lessons vary and some hurt, indicating content specificity rather than a pure length or style artifact. The circular step is confined to the poolC result: the lessons' source and the evaluation set coincide, so the poolC gain is in-sample. There are no self-citations or imported uniqueness theorems. Separate non-circularity threats include the absence of a matched control prompt, which weakens causal attribution of the gains to the specific error categories, and Table 3's confidence distribution suggesting the poolC-with-comment 80% filter should retain no errors, making the provenance of the 100 reviewed examples unclear; these are validity concerns, not derivation-circularity. Overall, partial circularity on poolC, with independent support from avatar.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper makes no formal derivation, so the ledger records the empirical premises the central claim depends on. The key dependencies are: a post hoc confidence cutoff of 80 for selecting which mistakes to analyze; unverified self-reported rationales as the basis for the error taxonomy; and representativeness of a 100-example poolC sample for both datasets. No new physical or mathematical entities are introduced.

free parameters (1)
  • Confidence threshold for reliable incorrect predictions = 80
    Section 3.2.2: the threshold is justified only by the observed bimodal distribution of PaLM confidence values (0 or >=80), and it controls which incorrect predictions are analyzed to construct the taxonomy.
assumptions (3)
  • domain assumption High-confidence incorrect predictions (confidence >= 80) are reliable signals of genuine model error rather than hallucination artifacts.
    Section 3.2.2: the cutoff is chosen post hoc from observed confidence values, with no independent evidence linking high confidence to the true cause of the error.
  • domain assumption The model's natural-language explanations in response to the prompt in Figure 4 truthfully reveal its computational error causes.
    Section 3.2.3: all eight prompt-bias categories are derived from these self-reports; LLM self-explanations are known to be unfaithful and this is not validated.
  • domain assumption The 100 poolC incorrect predictions manually reviewed are representative of prompt-bias errors in both poolC and avatar datasets.
    Section 3.2.3 and Section 6 (Threats): the paper itself concedes that the categories may be confined to poolC characteristics and may not cover all possible prompt bias mistakes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models." pith.science (2026). https://pith.science/paper/UHAETMFM

@misc{pith2026250505679,
  author       = {Pith},
  title        = {Pith review of: From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHAETMFM}},
  note         = {Machine review of arXiv:2505.05679}
}
read the original abstract

The issue of clone code has persisted in software engineering, primarily because developers often copy and paste code segments. This common practice has elevated the importance of clone code detection, garnering attention from both software engineering researchers and industry professionals. Their collective concern arises from the potential negative impacts that clone code can have on software quality. The emergence of powerful Generative Large Language Models (LLMs) like ChatGPT has exacerbated the clone code problem. These advanced models possess code generation capabilities that can inadvertently create code clones. As a result, the need to detect clone code has become more critical than ever before. In this study, we assess the suitability of LLMs for clone code detection. Our results demonstrate that the Palm model achieved a high F1 score of 89.30 for the avatar dataset and 86.41 for the poolC dataset. A known issue with LLMs is their susceptibility to prompt bias, where the performance of these models fluctuates based on the input prompt provided. In our research, we delve deeper into the reasons behind these fluctuations and propose a framework to mitigate prompt bias for clone detection. Our analysis identifies eight distinct categories of prompt bias, and our devised approach leveraging these biases yields a significant improvement of up to 10.81% in the F1 score. These findings underscore the substantial impact of prompt bias on the performance of LLMs and highlight the potential for leveraging model errors to alleviate this bias.

Figures

Figures reproduced from arXiv: 2505.05679 by the authors.

Figure 1
Figure 1. Overall Approach propose a method to improve model’s performances using the prompt bias lessons. (E) We evaluate the proposed method. An outline of our methodology is presented in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Lastly, we picked the top-performing model in terms of F1 score. You are a helpful assistant that takes two pieces of code snippets and detects whether these two pieces of code snippets are clone code or not. The first code snippet is written in <language1>, and the second code snippet is written in <language2>. Here are the two code snippets: Code Snippet1: <code1>. Code Snippet2: <code2>. If you think any part of … view at source ↗
Figure 3
Figure 3. Confidence Extraction Prompt We employed this confidence level to indicate the reliability of model’s responses. Therefore, we retained only those incorrect predictions where the model exhibited 80% confidence in its responses, as these instances represented cases where the model made mistakes with a high degree of confidence. We call them reliable incorrect predictions. We picked 80% as a cutoff point because, as w… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Rational Extraction Prompt 3.3 Calculating Prevalence of Prompt Bias Mistake Category: As we have defined the prompt bias mistake categories, next we want to understand the prevalence of each category. We adopted a topic modeling tool named BERTopic [24] to calculate t…
Figure 5
Figure 5. Figure 5: Example of Operational Ambiguities category [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Example of Confusion Over Varied Approaches category [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Best Performing Prompt [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 47 canonical work pages

  1. [1]

    chatgpt, howpublished = https://chat.openai.com/, note = Accessed: 2023-09-23

    “chatgpt, howpublished = https://chat.openai.com/, note = Accessed: 2023-09-23. ”

    work page 2023

  2. [2]

    “figshare-replication package, howpublished = https://figshare.com/s/4e885d949f72dd4b1840, note = Accessed: 2023-9-

    work page 2023

  3. [3]

    “Midjourney, howpublished = https://www.midjourney.com/home/?callbackurl=%2fapp%2f, note = Accessed: 2023-09-

    work page 2023

  4. [4]

    OpenAI-models, howpublished = https://platform.openai.com/docs/models/gpt-3-5, note = Accessed: 2023-09-23

    “OpenAI-models, howpublished = https://platform.openai.com/docs/models/gpt-3-5, note = Accessed: 2023-09-23. ”

    work page 2023

  5. [5]

    PoolC-5-fold-clone-detection-600k-5fold, howpublished = https://huggingface.co/datasets/poolc/5-fold-clone- detection-600k-5fold, note = Accessed: 2023-09-23

    “PoolC-5-fold-clone-detection-600k-5fold, howpublished = https://huggingface.co/datasets/poolc/5-fold-clone- detection-600k-5fold, note = Accessed: 2023-09-23. ” , Vol. 1, No. 1, Article . Publication date: May 2018. From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models 19

    work page 2023

  6. [6]

    surveyMonkey-sample size calculator, howpublished = https://www.surveymonkey.com/mp/sample-size-calculator/, note = Accessed: 2023-09-23

    “surveyMonkey-sample size calculator, howpublished = https://www.surveymonkey.com/mp/sample-size-calculator/, note = Accessed: 2023-09-23. ”

    work page 2023

  7. [7]

    wasiahmad-AVATAR, howpublished = https://github.com/wasiahmad/avatar, note = Accessed: 2023-09-23

    “wasiahmad-AVATAR, howpublished = https://github.com/wasiahmad/avatar, note = Accessed: 2023-09-23. ”

    work page 2023

  8. [8]

    Black box fairness testing of machine learning models,

    A. Aggarwal, P. Lohia, S. Nagar, K. Dey, and D. Saha, “Black box fairness testing of machine learning models, ” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2019, pp. 625–635

    work page 2019

Show all 62 references
  1. [9]

    A systematic review on code clone detection,

    Q. U. Ain, W. H. Butt, M. W. Anwar, F. Azam, and B. Maqbool, “A systematic review on code clone detection, ”IEEE access, vol. 7, pp. 86 121–86 144, 2019

    2019

  2. [10]

    Artificial hallucinations in chatgpt: implications in scientific writing,

    H. Alkaissi and S. I. McFarlane, “Artificial hallucinations in chatgpt: implications in scientific writing, ”Cureus, vol. 15, no. 2, 2023

    2023

  3. [11]

    The adverse effects of code duplication in machine learning models of code,

    M. Allamanis, “The adverse effects of code duplication in machine learning models of code, ” inProceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software , 2019, pp. 143–153

    2019

  4. [12]

    Clone detection using abstract syntax trees,

    I. D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier, “Clone detection using abstract syntax trees, ” inProceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) . IEEE, 1998, pp. 368–377

    1998

  5. [13]

    Do the machine learning models on a crowd sourced platform exhibit bias? an empirical study on model fairness,

    S. Biswas and H. Rajan, “Do the machine learning models on a crowd sourced platform exhibit bias? an empirical study on model fairness, ” inProceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engine...

    2020

  6. [14]

    Multiple significance tests: the bonferroni method,

    J. M. Bland and D. G. Altman, “Multiple significance tests: the bonferroni method, ”Bmj, vol. 310, no. 6973, p. 170, 1995

    1995

  7. [15]

    Making fair ml software using trustworthy explanation,

    J. Chakraborty, K. Peng, and T. Menzies, “Making fair ml software using trustworthy explanation, ” inProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering , 2020, pp. 1229–1233

    2020

  8. [16]

    Software engineering for fairness: A case study with hyperparameter optimization,

    J. Chakraborty, T. Xia, F. M. Fahid, and T. Menzies, “Software engineering for fairness: A case study with hyperparameter optimization, ”arXiv preprint arXiv:1905.05786, 2019

    1905 arXiv

  9. [17]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scaling language modeling with pathways, ”arXiv preprint arXiv:2204.02311, 2022

    2022 arXiv

  10. [18]

    Sample size determination,

    R. B. Dell, S. Holleran, and R. Ramakrishnan, “Sample size determination, ”ILAR journal, vol. 43, no. 4, pp. 207–213, 2002

    2002

  11. [19]

    A language independent approach for detecting duplicated code,

    S. Ducasse, M. Rieger, and S. Demeyer, “A language independent approach for detecting duplicated code, ” inProceedings IEEE International Conference on Software Maintenance-1999 (ICSM’99). ’Software Maintenance for Business Change’(Cat. No. 99CB36360). IEEE, 1999, pp. 109–118

    1999

  12. [20]

    Functional code clone detection with syntax and semantics fusion learning,

    C. e. a. Fang, “Functional code clone detection with syntax and semantics fusion learning, ” inISSTA 2020. ACM, 2020, p. 516–527. [Online]. Available: https://doi.org/10.1145/3395363.3397362

    2020

  13. [21]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages, ”arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  14. [22]

    An empirical examination of the impact of bias on just-in-time defect prediction,

    J. Gesi, J. Li, and I. Ahmed, “An empirical examination of the impact of bias on just-in-time defect prediction, ” in Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , 2021, pp. 1–12

    2021

  15. [23]

    News summarization and evaluation in the era of gpt-3,

    T. Goyal, J. J. Li, and G. Durrett, “News summarization and evaluation in the era of gpt-3, ”arXiv preprint arXiv:2209.12356, 2022

    2022 arXiv

  16. [24]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class-based tf-idf procedure, ” 2022

    2022

  17. [25]

    Computing inter-rater reliability and its variance in the presence of high agreement,

    K. L. Gwet, “Computing inter-rater reliability and its variance in the presence of high agreement, ”British Journal of Mathematical and Statistical Psychology , vol. 61, no. 1, pp. 29–48, 2008

    2008

  18. [26]

    The fickle p value generates irreproducible results,

    L. G. Halsey, D. Curran-Everett, S. L. Vowler, and G. B. Drummond, “The fickle p value generates irreproducible results, ” Nature methods, vol. 12, no. 3, pp. 179–185, 2015

    2015

  19. [27]

    Is neuron coverage a meaningful measure for testing deep neural networks?

    F. Harel-Canada, L. Wang, M. A. Gulzar, Q. Gu, and M. Kim, “Is neuron coverage a meaningful measure for testing deep neural networks?” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineer...

    2020

  20. [28]

    How good are gpt models at machine translation? a comprehensive evaluation,

    A. Hendy, M. Abdelrehim, A. Sharaf, V. Raunak, M. Gabr, H. Matsushita, Y. J. Kim, M. Afify, and H. H. Awadalla, “How good are gpt models at machine translation? a comprehensive evaluation, ”arXiv preprint arXiv:2302.09210, 2023

    2023 arXiv

  21. [29]

    How secure is code generated by chatgpt?

    R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How secure is code generated by chatgpt?” arXiv preprint arXiv:2304.09655, 2023

    2023 arXiv

  22. [30]

    T test as a parametric statistic,

    T. K. Kim, “T test as a parametric statistic, ”Korean journal of anesthesiology, vol. 68, no. 6, pp. 540–546, 2015

    2015

  23. [31]

    Using slicing to identify duplication in source code,

    R. Komondoor and S. Horwitz, “Using slicing to identify duplication in source code, ” inInternational static analysis symposium. Springer, 2001, pp. 40–56

    2001

  24. [32]

    Identifying similar code with program dependence graphs,

    J. Krinke, “Identifying similar code with program dependence graphs, ” inProceedings Eighth Working Conference on Reverse Engineering. IEEE, 2001, pp. 301–309. , Vol. 1, No. 1, Article . Publication date: May 2018. 20 Trovato et al

    2001

  25. [33]

    Bigclonebench considered harmful for machine learning,

    J. Krinke and C. Ragkhitwetsagul, “Bigclonebench considered harmful for machine learning, ” in 2022 IEEE 16th International Workshop on Software Clones (IWSC) . IEEE, 2022, pp. 1–7

    2022

  26. [34]

    A mathematical investigation of hallucination and creativity in gpt models,

    M. Lee, “A mathematical investigation of hallucination and creativity in gpt models, ”Mathematics, vol. 11, no. 10, p. 2320, 2023

    2023

  27. [35]

    Clorifi: software vulnerability discovery using code clone verification,

    H. Li, H. Kwon, J. Kwon, and H. Lee, “Clorifi: software vulnerability discovery using code clone verification, ”Concurrency and Computation: Practice and Experience , vol. 28, no. 6, pp. 1900–1917, 2016

    1900

  28. [36]

    Improving chatgpt prompt for code generation,

    C. Liu, X. Bao, H. Zhang, N. Zhang, H. Hu, X. Zhang, and M. Yan, “Improving chatgpt prompt for code generation, ” arXiv preprint arXiv:2305.08360, 2023

    2023 arXiv

  29. [37]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues,

    Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.-B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing and mitigating code quality issues, ”arXiv preprint arXiv:2307.12596, 2023

    2023 arXiv

  30. [38]

    Déjàvu: a map of code duplicates on github,

    C. V. Lopes, P. Maj, P. Martins, V. Saini, D. Yang, J. Zitny, H. Sajnani, and J. Vitek, “Déjàvu: a map of code duplicates on github, ”Proceedings of the ACM on Programming Languages , vol. 1, no. OOPSLA, pp. 1–28, 2017

    2017

  31. [39]

    hdbscan: Hierarchical density based clustering

    L. McInnes, J. Healy, and S. Astels, “hdbscan: Hierarchical density based clustering. ”J. Open Source Softw., vol. 2, no. 11, p. 205, 2017

    2017

  32. [40]

    Umap: Uniform manifold approximation and projection for dimension reduction,

    L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction, ” arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  33. [41]

    Ablation studies in artificial neural networks,

    R. Meyes, M. Lu, C. W. de Puiseau, and T. Meisen, “Ablation studies in artificial neural networks, ” arXiv preprint arXiv:1901.08644, 2019

    1901 arXiv

  34. [42]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback, ”Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  35. [43]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners, ”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  36. [44]

    Toxic code snippets on stack overflow,

    C. Ragkhitwetsagul, J. Krinke, M. Paixao, G. Bianco, and R. Oliveto, “Toxic code snippets on stack overflow, ”IEEE Transactions on Software Engineering , vol. 47, no. 3, pp. 560–581, 2019

    2019

  37. [45]

    Comparison and evaluation of code clone detection techniques and tools: A qualitative approach,

    C. K. Roy, J. R. Cordy, and R. Koschke, “Comparison and evaluation of code clone detection techniques and tools: A qualitative approach, ”Science of computer programming , vol. 74, no. 7, pp. 470–495, 2009

    2009

  38. [46]

    A survey on software clone detection research,

    C. K. Roy and J. R. Cordy, “A survey on software clone detection research, ”Queen’s School of computing TR , vol. 541, no. 115, pp. 64–68, 2007

    2007

  39. [47]

    Code clones: Detection and management,

    N. Saini, S. Singh et al., “Code clones: Detection and management, ”Procedia computer science, vol. 132, pp. 718–727, 2018

    2018

  40. [48]

    Generalizability of code clone detection on codebert,

    T. Sonnekalb, B. Gruner, C.-A. Brust, and P. Mäder, “Generalizability of code clone detection on codebert, ” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–3

    2022

  41. [49]

    Basics of qualitative research techniques,

    A. Strauss and J. Corbin, “Basics of qualitative research techniques, ” 1998

    1998

  42. [50]

    Code clone detection based on order and content of control statements,

    M. Sudhamani and L. Rangarajan, “Code clone detection based on order and content of control statements, ” in2016 2nd International Conference on Contemporary Computing and Informatics (IC3I) . IEEE, 2016, pp. 59–64

    2016

  43. [51]

    Testing dnn image classifiers for confusion & bias errors,

    Y. Tian, Z. Zhong, V. Ordonez, G. Kaiser, and B. Ray, “Testing dnn image classifiers for confusion & bias errors, ” in Proceedings of the acm/ieee 42nd international conference on software engineering , 2020, pp. 1122–1134

    2020

  44. [52]

    Detecting code clones with graph neural network and flow-augmented abstract syntax tree,

    W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural network and flow-augmented abstract syntax tree, ” in2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2020, pp. 261–271

    2020

  45. [53]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models, ”arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  46. [54]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, ”arXiv preprint arXiv:2109.00859, 2021

    2021 arXiv

  47. [55]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models, ”Advances in Neural Information Processing Systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  48. [56]

    Dombert: Domain-oriented language model for aspect-based sentiment analysis,

    H. Xu, B. Liu, L. Shu, and P. S. Yu, “Dombert: Domain-oriented language model for aspect-based sentiment analysis, ” arXiv preprint arXiv:2004.13816, 2020

    2004 arXiv

  49. [57]

    Graph-based code semantics learning for efficient semantic code clone detection,

    D. Yu, Q. Yang, X. Chen, J. Chen, and Y. Xu, “Graph-based code semantics learning for efficient semantic code clone detection, ”Information and Software Technology, vol. 156, p. 107130, 2023

    2023

  50. [58]

    Cloning practices: Why developers clone and what can be changed,

    G. Zhang, X. Peng, Z. Xing, and W. Zhao, “Cloning practices: Why developers clone and what can be changed, ” in2012 28th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 2012, pp. 285–294

    2012

  51. [59]

    A survey of software clone detection from security perspective,

    H. Zhang and K. Sakurai, “A survey of software clone detection from security perspective, ”IEEE Access, vol. 9, pp. 48 157–48 173, 2021. , Vol. 1, No. 1, Article . Publication date: May 2018. From Bias To Improved Prompts: A Case Study of Bias Mitigation of Clone Detection Models 21

    2021

  52. [60]

    Sentiment analysis for software engineering: How far can pre-trained transformer models go?

    T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, and L. Jiang, “Sentiment analysis for software engineering: How far can pre-trained transformer models go?” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2020, pp. 70–80

    2020

  53. [61]

    Deepsim: deep learning code functional similarity,

    G. Zhao and J. Huang, “Deepsim: deep learning code functional similarity, ” inProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2018, pp. 141–151

    2018

  54. [62]

    Large language models are human-level prompt engineers,

    Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers, ”arXiv preprint arXiv:2211.01910, 2022. , Vol. 1, No. 1, Article . Publication date: May 2018

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.