Pith. sign in

REVIEW 3 major objections 6 minor 22 references

Metamorphic Testing of Deep Code Models: A Systematic Literature Review

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This systematic review of 45 studies maps — for the first time, it argues — how metamorphic testing is applied to deep code models, finding the practice concentrated on identifier renaming and dead-code insertion.

desk verdict A genuinely useful first systematic review of metamorphic testing for deep-code models, with a strong taxonomy and a fixable but real corpus-validity problem that needs refereeing. read the letter →

arxiv 2507.22610 v1 pith:J2J5ZU5M submitted 2025-07-30 cs.SE cs.AI

Ali Asgari , Milan de Koning , Pouria Derakhshanfar , Annibale Panichella This is my paper · ORCID
classification cs.SEcs.AI
keywords metamorphictestingdeepcodemodelsLLM4Coderobustnessevaluationsemantic-preservingtransformationssystematicliteraturereviewadversarialintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims to be the first comprehensive, data-driven systematic review of metamorphic testing as applied to deep code models, the LLMs and classifiers that analyze or generate source code. From 45 primary studies, the authors build a taxonomy of 121 semantic-preserving transformations and map which models, tasks, languages, datasets, and metrics the field actually uses. The key finding is a lopsided practice: identifier renaming (86.5% of studies), dead-code insertion (64.5%), and control-flow tweaks dominate, while API-level rewrites, comment edits, and function restructuring are nearly absent. The authors argue this imbalance creates blind spots, because the easy-to-automate transformations are not the ones that expose deeper semantic weaknesses, and they propose a research roadmap spanning task, language, model-architecture, and metric gaps.

What carries the argument

The carrying object is the metamorphic relation: a semantic-preserving transformation applied to an input program, under which a robust model's output should stay stable, so that consistency between original and transformed predictions serves as an implicit oracle. The paper operationalizes this by classifying every transformation found in the 45 primary studies into eight categories (trivial, identifier renaming, comment, data, control flow, function, API, dead-code insertion, plus miscellaneous) and by classifying how transformations are applied into six technique families. The taxonomy plus the technique classification carries the argument: the reported percentages and gap claims are computed from these two classification schemes.

What would settle it

Re-run the search protocol with the same seven research questions but an expanded query that also captures the terms 'perturbation', 'data augmentation', and 'semantic-preserving mutation', and extend the Google Scholar step with a snowballing pass over the included studies' reference lists; if the corpus grows materially beyond 45 papers and the reported frequencies (for example, identifier renaming at 86.5% of studies) shift appreciably, the review's landscape claims would need to be revised.

Watch

Extended reading notes

Core claim

The authors claim to close a gap in the literature by synthesizing the fragmented research on metamorphic testing for LLM4Code, where the same technique appears under labels as different as counterfactual examples, adversarial attacks, and obfuscation. Analyzing 45 papers published from 2019 to September 2024, they identify 121 distinct transformations across eight categories and report that identifier renaming (86.5% of studies), dead-code insertion (64.5%), and control-flow transformations (58%) dominate, while API and comment transformations appear in only 6% of studies. On the application side they distinguish six technique families — one-pass, evolutionary, gradient-based, sampling-based, transfer-based, and embedding-based — and show that the field concentrates on encoder-only understanding models (CodeBERT in 15 studies), on clone detection and method-name prediction, on Java and Python, on a handful of benchmark datasets, and on a fragmented set of over 60 metric labels. The contribution is not a new testing method but a map of the landscape, used to ground a thirteen-theme research roadmap.

Load-bearing premise

Every percentage and every proposed research direction rests on the assumption that the 45 selected studies fairly represent the full body of work on metamorphic testing for deep code models, which depends on a manual paper-selection step that is not fully reproducible.

Editorial extensions

If this is right

  • Robustness findings from existing studies cannot be assumed to transfer beyond the dominant pairings of transformation, task, and model, since identifier renaming is studied in 86.5% of papers while API and comment transformations appear in only 6%.
  • Practitioners should be cautious about trusting reported robustness of code models, because evaluations concentrate on encoder-only models like CodeBERT and on understanding tasks, leaving generative decoder-only LLMs and generation-oriented tasks substantially untested.
  • Metamorphic testing can be extended beyond robustness to security, privacy, explainability, efficiency, and usability, with concrete metamorphic relations sketched for each quality objective.
  • Data leakage between pretraining and benchmarks is a likely confound, and robustness testing with task-appropriate transformations should accompany accuracy reporting as a standard practice.
  • Releasing shared transformation corpora, semantic validators, and standard robustness splits would let the field compare approaches on equal footing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported percentages are even approximately right, the community's picture of code-model robustness is shaped less by what actually breaks models than by which transformations are cheap to implement; a testable prediction is that adding API-level and comment transformations would reveal failure modes that identifier renaming misses.
  • The review's own terminology problem — that metamorphic testing hides under labels like perturbation and augmentation — suggests the true volume of relevant work may be larger than 45 studies, so the landscape percentages should be read as a lower-bound snapshot.
  • A direct extension the authors only gesture at: metamorphic relations defined over natural-language prompts, rather than code alone, would be the natural way to bring closed chatbot-style assistants such as GitHub Copilot and ChatGPT into robustness evaluation.
  • The connection drawn to mutation testing, treating metamorphic variants as equivalent mutants, hints at a practical tooling path that reuses existing mutation-testing infrastructure for robustness evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript reports a systematic literature review (SLR) of metamorphic testing (MT) applied to deep code models. Following the Kitchenham and Webster guidelines, the authors searched four databases plus Google Scholar, applied explicit inclusion and exclusion criteria, and selected 45 primary studies published between 2019 and September 2024. The review classifies 121 code transformations into a taxonomy, groups application techniques (one-pass, evolutionary, gradient-based, sampling-based, transfer-based, embedding-based), and summarizes the tasks, models, languages, datasets, and metrics used in robustness evaluations. The paper claims to be the first comprehensive, data-driven overview of MT for code-focused LLMs and concludes with a thirteen-item research roadmap. The stated main findings include identifier renaming in 86.5% of studies, dead code insertion in 64.5%, clone detection and method name prediction as the most common tasks (29% each), and CodeBERT as the most frequently tested model.

Significance. If the corpus and classifications are sound, this review would be a valuable contribution to the software testing and AI4Code communities. Its useful elements include a structured taxonomy of transformations, an explicit methodology with inter-rater disagreement resolution, a publicly available replication artifact, and a research roadmap grounded in observed coverage gaps. The review also makes a credible case that existing surveys either predate LLMs, cover adversarial robustness more broadly, or do not focus specifically on metamorphic testing for code models. However, the significance of the landscape claims depends directly on the correctness and scope of the 45-paper corpus and on the internal consistency of the reported statistics. Those issues are not peripheral: they affect every RQ-level percentage and the central claim that all 45 studies employ MT for robustness testing.

major comments (3)
  1. [§3.2, Table 2; §4.2.1; §2.3] The corpus appears to violate the paper's own inclusion criteria. Table 2 excludes 'papers focusing solely on data augmentation' and requires that 'modified code snippets are used for robustness evaluations of deep code models.' Yet §4.2.1 states that Jin et al. [67] used fixed transformations 'to generate augmented data sets for contrastive pre-training,' and that Liu et al. [91] stochastically applied transformations 'to generate semantically equivalent code variants.' These are descriptions of contrastive pre-training or data augmentation, not robustness evaluation. This directly contradicts §2.3, which asserts that 'all 45 primary studies analyzed employ MT to test models' resilience to input perturbations or semantic-preserving transformations.' Because headline percentages (e.g., identifier renaming in 86.5% of studies, clone detection at 29%) are computed over the 45-paper corpus, the inclusion of [67] and [91] inflates the denominator and biases every RQ result. The authors must either demonstrate that these two studies actually perform robustness evaluation (e.g., report robustness-specific metrics under perturbation) or remove them and recompute all frequency claims in Tables 5–11 and the RQ answers.
  2. [§4.7, Table 11] The evaluation-metric statistics are internally inconsistent. The text states that F1 appears in '12 of 45 studies (26.6%)', but Table 11 lists 16 references for F1 and reports a fraction of 35.56%. The text also calls Accuracy the second most reported metric at 28.89%, yet F1 at 26.6% would be a different ordering than the table's 35.56% versus 28.89%. Additionally, the text mentions 'over 60 distinct metric labels' while Table 11 enumerates fewer than 30 rows. These discrepancies are load-bearing because RQ7's main conclusions about metric fragmentation and metric prevalence rest on counts and percentages that the manuscript does not currently support. All counts, percentages, and the claimed number of distinct metric labels need to be reconciled.
  3. [§3.4, Table 4; §6] The Google Scholar component of the search protocol is not reproducible and could materially affect the reported frequencies. The paper states that the query produced thousands of hits and that 'we manually selected 20 additional papers' that met the inclusion criteria, without specifying the exact query used, the stopping rule, the screening procedure, or the list of papers considered and rejected. The authors acknowledge in §6 that 'some relevant studies may still have been missed due to naming variations and limited indexing,' but the manual selection of nearly half of the final corpus is a more serious threat than the acknowledged terminological ambiguity. If studies using terms such as 'perturbation' or 'augmentation' were over- or under-represented in that manual step, the percentages in RQ1–RQ7 could change substantially. The authors should document the Google Scholar search and screening in enough detail to allow replication, or provide a sensitivity analysis showing that the main conclusions are robust to the manual selection step.
minor comments (6)
  1. [§4.1, Table 5] Several percentages in the text do not match Table 5. For example, §4.1.3 says comment transformations were applied in 'about 9%' of studies, while Table 5 reports 6%; §4.1.4 says data transformations were used in 45.5% of studies, while Table 5 reports 50%. These values should be harmonized throughout Section 4.1.
  2. [§4.3, Table 6] The stated task fractions are inconsistent with the denominators. With N=45, 13 papers is 28.9% rather than 29%, 6 papers is 13.3% rather than 13.5%, 3 papers is 6.7% rather than 6.5%, and 2 papers is 4.4% rather than 4.5%. The rounding should be applied consistently across the table and the text.
  3. [§4.4, Table 8] The model-architecture percentages in the text (62.2%, 42.2%, 26.7%) are not clearly derivable from the table, since models can appear in multiple categories and the relationship between 'unique studies' and model counts is unspecified. Please define the denominator used for these percentages.
  4. [§4.6, Table 10] There is a duplicated phrase in the java-large discussion: 'to evaluate the robustness of the modelthe model against larger and more complex codebases' should read 'to evaluate the robustness of the model against larger and more complex codebases.'
  5. [§5.5.6] The text contains a typographical error: 'based on a empiricial study by Zhang et al.' should be 'based on an empirical study by Zhang et al.'
  6. [§2.2] The model name is misspelled as 'ChatGP' in the list of general-purpose LLMs; it should be 'ChatGPT.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SLR's taxonomy and frequency claims are external summaries of a manually collected corpus, and the authors' own earlier papers are included as primary studies without being load-bearing.

full rationale

This is a systematic literature review, not a derivation with fitted parameters or a predictive model. The central outputs—the 121-transformation taxonomy, the per-category usage percentages, and the rankings of tasks, models, languages, datasets, and metrics—are descriptive summaries of the 45 screened primary studies, obtained by manual classification with reported inter-rater disagreement resolution (Section 3.5). No equation or fitted parameter is used, so no 'prediction' can reduce to an input by construction. The authors do cite their own prior work ([8], [9], and [12] in the reference list) and include [8] and [9] among the primary studies, but none of the survey's claims depends on those self-citations for validity; the corpus percentages and taxonomy are aggregated over all 45 studies and the roadmap is derived from observed gaps, not from the authors' prior results. The statement in Section 2.3 that all 45 primary studies employ MT to test resilience is essentially a restatement of the inclusion criteria in Table 2, which require modifications to be used for robustness evaluations; this is a scope definition rather than an empirical discovery, and the paper explicitly frames its focus as robustness (Sections 1 and 3.2). The skeptical concern that some included contrastive-learning studies may not actually evaluate robustness is a corpus-validity or correctness issue, not a circularity issue: even if [67] or [91] were misclassified, the review would be inaccurate, but the inaccuracy would not consist of deriving its conclusions from those conclusions themselves. The threats-to-validity section acknowledges possible undercoverage and classification subjectivity, which further confirms that the authors treat the corpus as externally gathered evidence rather than as an assumption to be proved. Therefore, no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no fitted parameters, no new mathematical axioms, and no invented entities. Its numeric outputs are descriptive counts collected from the primary studies. The load-bearing assumptions are the representativeness of the literature corpus and the reliability of manual classification.

assumptions (2)
  • domain assumption The 45 primary studies identified by the search protocol constitute a representative and sufficiently complete sample of research on metamorphic testing for deep code models.
    Section 3.4 describes the database search and manual Google Scholar follow-up. All frequency claims and gap analysis depend on this corpus being representative. The threats to validity section acknowledges limited corpus size and possible undercoverage.
  • domain assumption Manual classification by the first two authors, with consensus discussion, accurately labels transformations, techniques, tasks, models, languages, datasets, and metrics.
    Section 3.5 states that 17 disagreements were resolved through discussions among all authors. Section 6 acknowledges the possibility of subjectivity in classification. The central taxonomy and statistics rely on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Metamorphic Testing of Deep Code Models: A Systematic Literature Review." pith.science (2026). https://pith.science/paper/J2J5ZU5M

@misc{pith2026250722610,
  author       = {Pith},
  title        = {Pith review of: Metamorphic Testing of Deep Code Models: A Systematic Literature Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2J5ZU5M}},
  note         = {Machine review of arXiv:2507.22610}
}
read the original abstract

Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models' robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.

Figures

Figures reproduced from arXiv: 2507.22610 by the authors.

Figure 1
Figure 1. Metamorphic Testing for Deep Code Models review: all 45 primary studies analyzed employ MT to test models’ resilience to input perturbations or semantic-preserving transformations, rather than to assess fairness or other attributes. The broader potential of MT to support quality evaluation beyond robustness remains underexplored, and we encourage future work to investigate how MT techniques can be extended or adapte… view at source ↗
Figure 2
Figure 2. Metamorphic Testing for Deep Code Models Framework the types of metamorphic transformations (e.g., renaming variables, inserting dead code) in RQ1, followed by an exploration of transformation application techniques (e.g., one-pass strategies or search-based methods) in RQ2. RQs 3 through 6 examine the broader evaluation context, including the targeted tasks (e.g., summarization, defect detection), deep code models … view at source ↗
Figure 3
Figure 3. Overview of publication years and publication types [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Publishing Venues [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Methodology Overflow which had a large result set. From this set, we manually selected 20 additional papers that met our inclusion criteria, leading to a !nal total of 45 primary studies. While our search strategy was comprehensive, we acknowledge that some relevant st…
Figure 6
Figure 6. Figure 6: Top Metrics and Transformation Types [PITH_FULL_IMAGE:figures/full_fig_p034_6.png]
Figure 7
Figure 7. Figure 7: Top Tasks and Transformation Types 5.1.2 Interpreting Transformation Preferences. We observe a clear preference for transformation types that are simpler to implement and less likely to introduce semantic ambiguity. For example, Identi!er Renaming appears in nearly 87%…
Figure 8
Figure 8. Figure 8: Top Models and Transformation Types complexity of ensuring semantic equivalence, particularly in cases where external dependencies or dynamic behaviors are involved. API modi!cations often require contextual understanding of call signatures and usage contracts, making …
Figure 9
Figure 9. Figure 9: Distribution of programming languages, IDEs, development technologies, and implemented functions used with GitHub Copilot, based on a empiricial study by Zhang et al. [120]. and Language Diversity), aligning test coverage with real-world language preferences enhances t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages

  1. [1]

    Examining zero-shot vulnerability repair with large language models

    [1]Hammond Pearce et al. “Examining zero-shot vulnerability repair with large language models”. In:2023 IEEE Symposium on Security and Privacy (SP). IEEE. 2023, pp. 2339–2356. J. ACM, Vol. 37, No. 4, Article

    work page 2023

  2. [4]

    Asleep at the keyboard? assessing the security of github copilot’s code contributions

    [13]Hammond Pearce et al. “Asleep at the keyboard? assessing the security of github copilot’s code contributions”. In:2022 IEEE Symposium on Security and Privacy (SP). IEEE. 2022, pp. 754–768. [14]Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. “Security and privacy challenges of large language models: A survey”. In:ACM Computing Surveys(2024). [15]Zhou...

    work page 2024

  3. [5]

    Search-based software test data generation: a survey

    111:46 Asgari et al. [17]Phil McMinn. “Search-based software test data generation: a survey”. In:Software testing, Veri!cation and reliability14.2 (2004), pp. 105–156. [18]Earl T. Barr et al. “The Oracle Problem in Software Testing: A Survey”. In:IEEE Transactions on Software Engineering41.5 (May 2015). Conference Name: IEEE Transactions on Software Engin...

    work page arXiv 2004

  4. [6]

    2000, pp

    Springer. 2000, pp. 849–858. J. ACM, Vol. 37, No. 4, Article

    work page 2000

  5. [7]

    #: 10.48550/arXiv.2109.00859. $%&: http://arxiv.org/abs/2109.00859 (visited on 10/30/2024). [34]Vincent J. Hellendoorn et al. “Deep learning type inference

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:47 [33]Yue Wang et al.CodeT5: Identi!er-aware Uni!ed Pre-trained Encoder-Decoder Models for Code Understanding and Generation. arXiv:2109.00859. Sept. 2021.!"#: 10.48550/arXiv.2109.00859. $%&: http://arxiv.org/abs/2109.00859 (visited on 10/30/2024). [34]Vincent J. Hellendoorn et a...

    arXiv doi:10.48550/arxiv.2109.00859 2021

  6. [9]

    Black-box testing of deep neural networks through test case diversity

    111:48 Asgari et al. [49]Zohreh Aghababaeyan et al. “Black-box testing of deep neural networks through test case diversity”. In:IEEE Transactions on Software Engineering49.5 (2023), pp. 3182–3204. [50]Zhen Li et al. “A comparative study of adversarial training methods for neural models of source code”. In:Future Generation Computer Systems142 (May 2023), ...

    work page doi:10.1016/j.future.2022.12.030 2023

  7. [10]

    A Closer Look into Transformer-Based Code Intelligence Through Code Transformation: Challenges and Opportunities

    48550/arXiv.2207.04285.$%&: http://arxiv.org/abs/2207.04285 (visited on 09/16/2024). [94]Zhen Li et al.Towards Making Deep Learning-based Vulnerability Detectors Robust. arXiv:2108.00669 [cs]. Aug. 2021.!"#: 10.48550/arXiv.2108.00669.$%&: http://arxiv.org/abs/2108.00669 (visited on 09/16/2024). [95]Heng Li et al.Black-box Adversarial Example Attack toward...

    work page Pith review arXiv 2024

  8. [11]

    Code Di"erence Guided Adversarial Example Genera- tion for Deep Code Models

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:49 [63]Zhao Tian, Junjie Chen, and Zhi Jin. “Code Di"erence Guided Adversarial Example Genera- tion for Deep Code Models”. In:2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). ISSN: 2643-1572. Sept. 2023, pp. 850–862.!"#: 10.1109/ASE56229. 2023.0...

    work page arXiv 2024

Show all 22 references
  1. [12]

    #: 10.48550/arXiv.2308.03312.$%&: http://arxiv.org/abs/2308.03312 (visited on 09/17/2024). [77]Xi Ding et al. “Adversarial Attack and Robustness Improvement on Code Summarization

    111:50 Asgari et al. [76]Kexin Pei et al.Exploiting Code Symmetries for Learning Program Semantics. arXiv:2308.03312 [cs]. Sept. 2024.!"#: 10.48550/arXiv.2308.03312.$%&: http://arxiv.org/abs/2308.03312 (visited on 09/17/2024). [77]Xi Ding et al. “Adversarial Attack and Robustn...

    2024 doi arXiv

  2. [13]

    On the generalizability of Neural Program Models with respect to semantic-preserving program transformations

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:51 [88]Zongjie Li et al.CCTEST: Testing and Repairing Code Completion Systems. arXiv:2208.08289 [cs]. May 2023.!"#: 10.48550/arXiv.2208.08289.$%&: http://arxiv.org/abs/2208.08289 (visited on 09/16/2024...

    2023 doi arXiv

  3. [15]

    Deceiving neural source code classi!ers: !nding adversarial examples with grammatical evolution

    48550/arXiv.2303.08509.$%&: http://arxiv.org/abs/2303.08509 (visited on 09/18/2024). [96]Claudio Ferretti and Martina Saletta. “Deceiving neural source code classi!ers: !nding adversarial examples with grammatical evolution”. In:Proceedings of the Genetic and Evo- lutionary Co...

    2024 arXiv

  4. [17]

    Monte Carlo tree search: A review of recent modi!cations and applications

    111:52 Asgari et al. [99]Maciej %wiechowski et al. “Monte Carlo tree search: A review of recent modi!cations and applications”. In:Arti!cial Intelligence Review56.3 (2023), pp. 2497–2562. [100]Je"rey Svajlenko and Chanchal K. Roy. “Evaluating clone detection tools with BigClon...

    2023

  5. [19]

    Source code authorship attribution using long short-term memory based networks: 22nd European Symposium on Research in Computer Security, ESORICS 2017

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:53 Association for Computing Machinery, Oct. 2016, pp. 731–747.#’(): 978-1-4503-4444-9. !"#: 10.1145/2983990.2984041.$%&: https://dl.acm.org/doi/10.1145/2983990.2984041 (visited on 10/30/2024). [114]Ba...

    2017

  6. [20]

    DeepSeek-Coder: When the Large Language Model Meets Programming– The Rise of Code Intelligence

    111:54 Asgari et al. [131]Daya Guo et al. “DeepSeek-Coder: When the Large Language Model Meets Programming– The Rise of Code Intelligence”. In:arXiv preprint arXiv:2401.14196(2024). [132]Yuxiang Wei et al. “Magicoder: Empowering code generation with oss-instruct”. In:arXiv pre...

    2024 arXiv

  7. [21]

    What are we really testing in mutation testing for machine learning? a critical re$ection

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:55 [150]Annibale Panichella and Cynthia CS Liem. “What are we really testing in mutation testing for machine learning? a critical re$ection”. In:2021 IEEE/ACM 43rd International Conference on Software ...

    2013

  8. [32]

    SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair

    Curran Associates, Inc., 2019.$%&: https://proceedings.neurips. cc/paper_!les/paper/2019/hash/49265d2447bc3bbfe9e76306ce40a31f-Abstract.html (visited on 10/30/2024). [107]Wasi Uddin Ahmad et al.Uni!ed Pre-training for Program Understanding and Generation. arXiv:2103.06333. Apr...

    2023 doi arXiv

  9. [51]

    Diversevul: A new vulnerable source code dataset for deep learn- ing based vulnerability detection

    [3]Yizheng Chen et al. “Diversevul: A new vulnerable source code dataset for deep learn- ing based vulnerability detection”. In:Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses. 2023, pp. 654–668. [4]Fang Liu et al. “Multi-task le...

    2023

  10. [79]

    Analyzing the Past to Prepare for the Future: Writing a Literature Review

    [54]Jane Webster and Richard T. Watson. “Analyzing the Past to Prepare for the Future: Writing a Literature Review”. In:MIS Quarterly26.2 (2002). Publisher: Management Information Systems Research Center, University of Minnesota, pp. xiii–xxiii.#’’): 0276-7783.$%&: https://www...

    2002

  11. [111]

    Publication date: August 2025

    2025

  12. [279]

    Metamorphic testing: A review of challenges and opportunities

    [23]Tsong Yueh Chen et al. “Metamorphic testing: A review of challenges and opportunities”. In:ACM Computing Surveys (CSUR)51.1 (2018), pp. 1–27. [24]Sergio Segura and Zhi Quan Zhou. “Metamorphic testing 20 years later: A hands-on in- troduction”. In:Proceedings of the 40th In...

    2018 doi arXiv

  13. [2018]

    #: 10.1145/ 3236024 . 3236051.$%&: https : / / dl . acm . org / doi / 10 . 1145 / 3236024 . 3236051 (visited on 10/30/2024). [35]Josh Achiam et al. “Gpt-4 technical report

    New York, NY, USA: Association for Computing Machinery, Oct. 2018, pp. 152–162.#’(): 978-1-4503-5573-5.!"#: 10.1145/ 3236024 . 3236051.$%&: https : / / dl . acm . org / doi / 10 . 1145 / 3236024 . 3236051 (visited on 10/30/2024). [35]Josh Achiam et al. “Gpt-4 technical report”...

    2023 arXiv

  14. [2025]

    Large language model for vulnerability detection: Emerging results and future directions

    Metamorphic Testing of Deep Code Models: A Systematic Literature Review 111:45 [2]Xin Zhou, Ting Zhang, and David Lo. “Large language model for vulnerability detection: Emerging results and future directions”. In:Proceedings of the 2024 ACM/IEEE 44th Interna- tional Conference...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.