Pith. sign in

REVIEW 4 major objections 7 minor 42 references

A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CodeT5 generally outperforms CodeBERT at generating vulnerability patches across six datasets, while CodeBERT wins on fragmented code, and fine-tuning fails to generalize to unseen datasets.

desk verdict The paper's central claim that CodeT5 generally outperforms CodeBERT is an artifact of dataset counting: two of the six datasets are invalid, and the remaining four split 2-2. read the letter →

arxiv 2506.04987 v1 pith:NMDLW3AZ submitted 2025-06-05 cs.SE cs.AI

Zanis Ali Khan , Aayush Garg , Qiang Tang This is my paper
classification cs.SEcs.AI
keywords automatedprogramrepairvulnerabilitypatchingCodeBERTT5BLEUCrystalout-of-distributiongeneralizationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates two pre-trained code models, CodeBERT and CodeT5, as tools for automated vulnerability repair. It fine-tunes each model on six vulnerability datasets spanning Go, PHP, C, and Java, then measures patch quality with CodeBLEU and CrystalBLEU. The central claim is that CodeT5 generally produces better patches for complex vulnerability patterns, while CodeBERT holds up better on fragmented or sparse code contexts. The paper also shows that fine-tuning lifts in-distribution accuracy but neither model generalizes to out-of-distribution datasets, so cross-domain vulnerability patching remains unsolved. A sympathetic reader would care because these are concrete, comparative benchmarks for a security-critical task where model choice and generalization limits currently lack empirical grounding.

What carries the argument

The evaluation is carried by two pre-trained transformers: CodeBERT, an encoder-only model that captures code–language structure, and CodeT5, an encoder–decoder model pre-trained for code generation. Patch quality is measured by CodeBLEU, which combines n-gram overlap with AST and dataflow similarity, and CrystalBLEU, which discounts trivially shared n-grams. The experimental machinery is a per-dataset fine-tuning loop followed by cross-dataset transfer tests, with heatmaps comparing in-distribution and out-of-distribution scores.

What would settle it

One concrete check would be to compile and run each generated patch against the test suite or vulnerability exploit for its dataset entry; if patches with near-perfect CodeBLEU scores fail to fix the vulnerability or introduce new flaws, the paper's comparative ranking would not reflect real repair quality. A simpler version: re-score the generated patches with a static security analyzer and see whether CodeT5's advantage on Vul4J survives.

Watch

Extended reading notes

Core claim

The paper's central discovery is a dataset-dependent performance split between two architectures. After fine-tuning on each dataset, CodeT5 out-scores CodeBERT on Vul4J, CodeParrot, and both MegaVul C datasets, which the paper attributes to CodeT5's broader pre-training on diverse code; CodeBERT in turn out-scores CodeT5 on Go and PHP, which the paper attributes to those datasets containing incomplete functions or isolated snippets where CodeBERT's structure-focused fine-tuning adapts better. On unseen datasets, both models drop sharply, with the exception of same-language transfer such as MegaVul_C_2023 to MegaVul_C_2024, revealing that fine-tuning encodes dataset-specific signals rather than general security principles.

Load-bearing premise

The load-bearing premise is that high CodeBLEU and CrystalBLEU similarity between a generated patch and the dataset's labeled fixed code means the patch is actually secure and correct, an assumption the paper itself flags as possibly overestimating performance.

Editorial extensions

If this is right

  • For datasets with complex, multi-line vulnerability patterns, CodeT5 is the better default model, whereas CodeBERT is preferable when inputs are incomplete or context-poor.
  • Fine-tuning both models on a single dataset yields high in-distribution scores but does not transfer to unseen datasets, so practical deployment needs per-target retraining or stronger generalization techniques.
  • Same-language out-of-distribution transfer (e.g., MegaVul_C_2023 to MegaVul_C_2024) remains high, suggesting that language-adjacent datasets share enough structure for useful transfer.
  • The reported CodeBLEU and CrystalBLEU scores establish a benchmark for future vulnerability-repair models on these six datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the metrics measure similarity to the labeled fixed code rather than security, the ranking may not hold when patches are judged by whether they actually neutralize the CVE; an execution-based evaluation of the same outputs would test this directly.
  • The finding that CodeBERT wins on fragmented contexts suggests that lightweight encoder-only models may be a cheaper drop-in for patch suggestion in code snippets, a hypothesis the paper does not test.
  • If cross-dataset accuracy drops reflect dataset-specific overfitting, then augmenting fine-tuning data with nearby-language vulnerabilities, or multi-task training on all six datasets at once, could improve out-of-distribution transfer; this is a concrete next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents a multi-dataset evaluation of CodeBERT and CodeT5 for automated vulnerability repair. For each of six datasets (Go, PHP, MegaVul_C_2023, MegaVul_C_2024, Vul4J, and CodeParrot), the authors fine-tune both models and report CodeBLEU and CrystalBLEU scores. Their central claims are that CodeT5 generally outperforms CodeBERT, especially on complex vulnerability patterns, that CodeBERT performs comparatively better on fragmented or sparse contexts, and that fine-tuning improves in-distribution performance but fails to generalize out-of-distribution. These claims are supported only by single-run similarity-based metrics, and, as I detail below, two of the six dataset entries are not valid evidence for vulnerability repair.

Significance. If the reported comparison were valid, this paper would provide a useful benchmark for two widely used pre-trained code models in the relatively underexplored area of vulnerability-specific program repair. It also ships a publicly available artifact on Zenodo, which is commendable for reproducibility. However, the central 'generally outperforms' claim is currently an artifact of dataset counting: two of the six rows in Table 2 are duplicates of the same data, and one row is a general code corpus without vulnerability labels or paired fixes. The evaluation also relies entirely on BLEU-style similarity scores rather than functional or security correctness, a limitation the authors themselves acknowledge in Section 7. The paper's main value is as a starting point; the central comparative conclusions are not yet established by the reported evidence.

major comments (4)
  1. [§4.1, Tables 1 and 2] The dataset validity is load-bearing and fails for two of the six entries. MegaVul_C_2023 and MegaVul_C_2024 have identical row counts (17,975 before preprocessing, 14,526 after), identical preprocessing counts (Rtok=3,147, Rcomm=0, Rnorm=302), and identical CodeBLEU and CrystalBLEU scores in every cell of Table 2, indicating they are the same data duplicated under two labels. CodeParrot, cited to the Hugging Face github-code-clean corpus, is a general code corpus without vulnerability labels or paired vulnerable/fixed examples; its near-perfect scores (CodeBLEU ~0.997, CrystalBLEU ~0.960) suggest the model is copying the input to the output, contributing no signal about vulnerability repair. After removing these invalid entries, only Go, PHP, MegaVul (deduplicated), and Vul4J remain, with CodeBERT better on Go/PHP and CodeT5 better on MegaVul/Vul4J. The Section 5.1 conclusion that 'CodeT5 consistently outperforms CodeBERT in VUL4J and CodeParrot datasets' and the abstract's 'generally outperforms' claim are therefore not supported by the evidence.
  2. [Table 2 and §5.1] The table is labeled 'Accuracy Scores' and the text repeatedly refers to model 'accuracy,' but the reported values are CodeBLEU and CrystalBLEU, which measure n-gram overlap and AST/data-flow similarity, not whether a generated patch is correct or secure. Section 7 explicitly concedes that these metrics 'primarily gauge syntactic and limited semantic cues' and may 'overestimate model performance,' and that the labeled 'patched' instances may not be truly secure. Because the central comparative ranking is expressed in terms of accuracy, the paper overstates what its measurements establish. The claims should be rephrased as code-similarity-based comparisons, and ideally supplemented with a functional or security validation of a sample of generated patches.
  3. [§5.2] The paper acknowledges 'non-deterministic behavior' and 'sporadic variability across executions' but reports only single runs for every configuration, with no error bars, confidence intervals, or statistical tests. Consequently, differences such as CodeBERT 0.7641 vs. CodeT5 0.6499 on Go (Table 2) could be within run-to-run noise. The central comparative claims require repeated runs and a significance test (e.g., paired bootstrap or Wilcoxon) before they can be accepted.
  4. [§5.2, Figures 1 and 2] The out-of-distribution generalization analysis is undermined by the same dataset validity problems. The authors highlight that 'trained on MegaVul_C_2023 and tested on MegaVul_C_2024' retains high accuracy, but if these two are the same data, this is in-distribution evaluation, not cross-dataset generalization. Similarly, the statement that Vul4J and CodeParrot are 'both Java-based' and show transfer is not meaningful for vulnerability repair because CodeParrot lacks vulnerability labels and fixes. The OOD analysis should be redone on the deduplicated, valid dataset set.
minor comments (7)
  1. [Table 2] The header 'Accuracy Scores' should be replaced with 'CodeBLEU and CrystalBLEU scores' to match the actual columns and avoid misrepresenting the metrics as correctness measures.
  2. [Table 1] The abbreviations Irows and Trows are not defined in the caption. Please expand them to 'initial rows' and 'total rows after preprocessing' or explain in the caption.
  3. [§3.2] The text states that 'all overlapping or duplicate instances were excluded' to avoid data leakage, yet MegaVul_C_2023 and MegaVul_C_2024 appear to be identical datasets with the same row counts and preprocessing results. If duplicates were excluded, one of these entries should be empty; please clarify how this step was applied.
  4. [§4.1] The description of the datasets says all six contain 'vulnerable source code along with their corresponding fixed versions,' but CodeParrot is a general code corpus and does not contain vulnerability labels or paired fixes. Please correct this description for accuracy.
  5. [Abstract and §5.1] The word 'accuracy' is used throughout the abstract and results (e.g., 'evaluate their accuracy' and 'CodeT5 generally outperforms CodeBERT in accuracy') even though the paper only reports CodeBLEU and CrystalBLEU. Consider replacing 'accuracy' with 'similarity-based scores' or 'code-similarity performance.'
  6. [Figures 1 and 2] The heatmaps are difficult to interpret as printed; axis labels, color-bar ranges, and the meaning of rows/columns are not clearly legible. Please enlarge the fonts and explain in the caption what the rows and columns represent (e.g., training dataset vs. test dataset).
  7. [Throughout] There are a few typos and naming inconsistencies, e.g., 'VUL4J' vs. 'Vul4J', 'Megavul' vs. 'MegaVul', and 'These datasets comprises' should be 'comprise'. A careful proofread would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark against external datasets and metrics, with no derivation step that reduces to its own inputs.

full rationale

The paper's central claims are empirical comparisons of CodeBERT and CodeT5 on vulnerability repair, evaluated with CodeBLEU and CrystalBLEU against held-out test splits of six external datasets. The models are fine-tuned on 85% training splits and tested on the remaining 15% (Section 3.2), so the RQ1 results are standard in-distribution evaluation, not self-derivation. RQ2 similarly fine-tunes on one dataset and tests on other datasets, which is a genuine out-of-distribution evaluation. The metrics are established external measures (Section 4.3), and the models are publicly available pre-trained checkpoints, not quantities defined in terms of the reported outcomes. The paper's self-citations appear only as related-work context, preprocessing motivations, and the Zenodo artifact link; none of these carries a load-bearing premise that reduces a result to its own input. The acknowledged threats in Section 7 — that BLEU-style metrics gauge syntactic/limited semantic similarity and that labeled 'patched' instances may not be truly secure — are validity concerns about what the benchmark measures, not circular reasoning. Similarly, the skeptic's observations about CodeParrot being an unpaired corpus and MegaVul_C_2023/2024 appearing to be duplicates are data-quality and dataset-validity issues that affect whether the claimed ranking is supported, but they do not make any derivation circular. No equation, parameter, or claimed prediction is equivalent by construction to an input, so the appropriate circularity finding is none.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on dataset label quality, the validity of the evaluation metrics, and the safety of the preprocessing pipeline. No new entities are introduced. The single listed free parameter captures the unreported training choices that affect all results.

free parameters (1)
  • Fine-tuning hyperparameters = not reported
    The experimental results depend on learning rates, epochs, batch sizes, and other training choices, but the paper reports none of these values. This makes the comparison hard to assess and reproduce.
assumptions (3)
  • domain assumption The 'fixed' code in all six datasets is correct and secure.
    Section 4.1 treats dataset entries as ground truth, but Section 7 acknowledges labeled patches may not represent truly secure fixes.
  • domain assumption CodeBLEU and CrystalBLEU capture meaningful patch quality and security relevance.
    Section 4.3 defines these metrics as similarity measures, and Section 7 admits they rely on syntactic and limited semantic cues, potentially overestimating performance.
  • domain assumption Preprocessing steps such as token truncation and comment removal do not remove security-critical context.
    Section 3.1 describes these steps, while Section 7 acknowledges they may eliminate vital context needed for security-relevant patches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair." pith.science (2026). https://pith.science/paper/NMDLW3AZ

@misc{pith2026250604987,
  author       = {Pith},
  title        = {Pith review of: A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMDLW3AZ}},
  note         = {Machine review of arXiv:2506.04987}
}
read the original abstract

Software vulnerabilities pose significant security threats, requiring effective mitigation. While Automated Program Repair (APR) has advanced in fixing general bugs, vulnerability patching, a security-critical aspect of APR remains underexplored. This study investigates pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across six datasets and four languages. We evaluate their accuracy and generalization to unknown vulnerabilities. Results show that while both models face challenges with fragmented or sparse context, CodeBERT performs comparatively better in such scenarios, whereas CodeT5 excels in capturing complex vulnerability patterns. CodeT5 also demonstrates superior scalability. Furthermore, we test fine-tuned models on both in-distribution (trained) and out-of-distribution (unseen) datasets. While fine-tuning improves in-distribution performance, models struggle to generalize to unseen data, highlighting challenges in robust vulnerability detection. This study benchmarks model performance, identifies limitations in generalization, and provides actionable insights to advance automated vulnerability patching for real-world security applications.

Figures

Figures reproduced from arXiv: 2506.04987 by the authors.

Figure 1
Figure 1. Heatmaps for CodeBERT [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Heatmaps for CodeT5 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 17 canonical work pages

  1. [1]

    In: Cybersecurity Systems for Hu- man Cognition Augmentation, Advances in Information Security, vol

    Albanese, M., Çam, H., Jajodia, S.: Automated cyber situation awareness tools and models for improving analyst performance. In: Cybersecurity Systems for Hu- man Cognition Augmentation, Advances in Information Security, vol. 61, pp. 47–

    work page

  2. [2]

    In: 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC)

    Arusoaie, A., Ciobâca, S., Craciun, V., Gavrilut, D., Lucanu, D.: A comparison of open-source static analysis tools for vulnerability detection in c/c++ code. In: 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC). pp. 161–168 (2017).https://doi.org/10.1109/SYNASC. 2017.00035

    arXiv 2017

  3. [3]

    Bui, Q., Paramitha, R., Vu, D., Massacci, F., Scandariato, R.: Apr4vul: an em- pirical study of automatic program repair techniques on real-world java vul- nerabilities. Empir. Softw. Eng. 29(1), 18 (2024). https://doi.org/10.1007/ S10664-023-10415-7, https://doi.org/10.1007/s10664-023-10415-7

    doi:10.1007/s10664-023-10415-7 2024

  4. [4]

    In: 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR)

    Bui, Q.C., Scandariato, R., Ferreyra, N.E.D.: Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In: 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR). pp. 464–468 (2022).https://doi.org/10.1145/3524842.3528482

    arXiv 2022

  5. [5]

    In: 8th USENIX Conference on Operating Systems Design and Implementation

    Cadar, C., Dunbar, D., Engler, D.: Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In: 8th USENIX Conference on Operating Systems Design and Implementation. p. 209–224. OSDI’08, USENIX Association, USA (2008)

    work page 2008

  6. [6]

    https://doi.org/10.1109/TSE.2020.3020502

    Chakraborty, S., Ding, Y., Allamanis, M., Ray, B.: Codit: code editing with tree- basedneuralmodels.IEEETransactionsonSoftwareEngineering 48(4),1385–1399 (2018). https://doi.org/10.1109/TSE.2020.3020502

    arXiv 2018

  7. [7]

    IEEE Transactions on Software Engineering47(09), 1943–1959 (September 2021)

    Chen, Z., Kommrusch, S., Tufano, M., Pouchet, L.N., Poshyvanyk, D., Monper- rus, M.: Sequencer: sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering47(09), 1943–1959 (September 2021). https://doi.org/10.1109/TSE.2019.2940179

    arXiv 1943

  8. [8]

    Dang, N.N.H., Thanh, T.Q., Nguyen-Duc, A.: BERTVRepair: On the Adoption of CodeBERT for Automated Vulnerability Code Repair, pp. 173–196. Springer Na- ture Switzerland, Cham (2024).https://doi.org/10.1007/978-3-031-55642-5_ 8, https://doi.org/10.1007/978-3-031-55642-5_8

    work page doi:10.1007/978-3-031-55642-5_ 2024

Show all 42 references
  1. [9]

    de-Fitero-Dominguez, D., García-López, E., García-Cabot, A., Martínez-Herráiz, J.J.: Enhanced automated code vulnerability repair using large language mod- els. Eng. Appl. Artif. Intell. 138, 109291 (2024). https://doi.org/10.1016/J. ENGAPPAI.2024.109291, https://doi.org/10.10...

    2024

  2. [10]

    In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

    Ding, Y., Ray, B., Devanbu, P., Hellendoorn, V.J.: Patching as translation: the data and the metaphor. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. p. 275–286. ASE’20, Association for Computing Machinery, New York, NY, USA (2...

    2021

  3. [11]

    In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

    Eghbali, A., Pradel, M.: Crystalbleu: precisely and efficiently measuring the sim- ilarity of code. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. pp. 1–12 (2022)

    2022

  4. [12]

    arXiv preprint arXiv:2002.08155 (2020) 14 Zanis Ali Khan, Aayush Garg, and Qiang Tang

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020) 14 Zanis Ali Khan, Aayush Garg, and Qiang Tang

    2020 arXiv

  5. [13]

    In: Findings of the Association for Computational Lin- guistics: EMNLP 2020

    Feng, Z., Guo, D., Tang, D., et al.: Codebert: a pre-trained model for programming and natural languages. In: Findings of the Association for Computational Lin- guistics: EMNLP 2020. pp. 1536–1547. Association for Computational Linguistics (November 2020). https://doi.org/10.1...

    2020 doi

  6. [14]

    CoRR abs/2012.11701 (2020), https://arxiv.org/abs/2012.11701

    Garg, A., Degiovanni, R., Jimenez, M., Cordy, M., Papadakis, M., Traon, Y.L.: Learning to predict vulnerabilities from vulnerability-fixes: A machine translation approach. CoRR abs/2012.11701 (2020), https://arxiv.org/abs/2012.11701

    2020 arXiv

  7. [15]

    Garg, A., Degiovanni, R., Jimenez, M., Cordy, M., Papadakis, M., Traon, Y.L.: Learning from what we know: How to perform vulnerability prediction using noisy historical data. Empir. Softw. Eng.27(7), 169 (2022).https://doi.org/10.1007/ S10664-022-10197-4, https://doi.org/10.10...

    2022 doi

  8. [16]

    CoRR abs/2303.04247 (2023)

    Garg, A., Degiovanni, R., Papadakis, M., Traon, Y.L.: Vulnerability mimicking mu- tants. CoRR abs/2303.04247 (2023). https://doi.org/10.48550/ARXIV.2303. 04247, https://doi.org/10.48550/arXiv.2303.04247

    2023 doi arXiv

  9. [17]

    In: IEEE Con- ference on Software Testing, Verification and Validation, ICST 2024, Toronto, ON, Canada, May 27-31, 2024

    Garg, A., Degiovanni, R., Papadakis, M., Traon, Y.L.: On the coupling between vulnerabilities and llm-generated mutants: A study on vul4j dataset. In: IEEE Con- ference on Software Testing, Verification and Validation, ICST 2024, Toronto, ON, Canada, May 27-31, 2024. pp. 305–3...

    2024

  10. [18]

    techrxiv preprint (Dec 2024).https:// doi.org/10.36227/techrxiv.173385946.65994728/v1, http://dx.doi.org/10

    Garg, A., Patsakis, C., Khan, Z.A., Tang, Q.: Payload analysis of adversaries’ tool- ing: Automated identification of fuzzers. techrxiv preprint (Dec 2024).https:// doi.org/10.36227/techrxiv.173385946.65994728/v1, http://dx.doi.org/10. 36227/techrxiv.173385946.65994728/v1

    2024

  11. [19]

    Gharibi, R., Sadreddini, M.H., Fakhrahmad, S.M.: T5APR: empowering au- tomated program repair across languages through checkpoint ensemble. J. Syst. Softw. 214, 112083 (2024).https://doi.org/10.1016/J.JSS.2024.112083, https://doi.org/10.1016/j.jss.2024.112083

    2024

  12. [20]

    Commu- nications of the ACM62(12), 56–65 (2019).https://doi.org/10.1145/3318162

    Goues, C.L., Pradel, M., Roychoudhury, A.: Automated program repair. Commu- nications of the ACM62(12), 56–65 (2019).https://doi.org/10.1145/3318162

    2019 doi

  13. [21]

    In: 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021

    Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svy- atkovskiy, A., Fu, S., Tufano, M., Deng, S.K., Clement, C.B., Drain, D., Sundare- san, N., Yin, J., Jiang, D., Zhou, M.: Graphcodebert: Pre-training code represen- tations with data flow. In: 9th I...

    2021

  14. [22]

    In: European Symposium on Research in Computer Security

    Guo, Y., Hu, Q., Tang, Q., Traon, Y.L.: An empirical study of the imbalance issue in software vulnerability detection. In: European Symposium on Research in Computer Security. pp. 371–390. Springer (2023)

    2023

  15. [23]

    In: Guide to Vulnerability Analysis for Com- puter Networks and Systems - An Artificial Intelligence Approach, pp

    Khan, S., Parkinson, S.: Review into state of the art of vulnerability assess- ment using artificial intelligence. In: Guide to Vulnerability Analysis for Com- puter Networks and Systems - An Artificial Intelligence Approach, pp. 3–32. Computer Communications and Networks, Spr...

    2018 doi

  16. [24]

    15599983

    Khan, Z.A., Garg, A., Tang, Q.: Artifact for a multi-dataset evaluation of mod- els for automated vulnerability repair (2025).https://doi.org/10.5281/zenodo. 15599983

    2025 doi

  17. [25]

    In: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022

    Khan, Z.A., Shin, D., Bianculli, D., Briand, L.C.: Guidelines for assessing the accuracy of log message template identification techniques. In: 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. pp. 1095–1106. ...

    2022

  18. [26]

    Khan, Z.A., Shin, D., Bianculli, D., Briand, L.C.: Impact of log parsing on deep learning-based anomaly detection. Empir. Softw. Eng. 29(6), 139 (2024). https://doi.org/10.1007/S10664-024-10533-W, https://doi.org/10. 1007/s10664-024-10533-w

    2024 doi

  19. [28]

    In: 34th In- ternational Conference on Software Engineering (ICSE)

    Le Goues, C., Dewey-Vogt, M., Forrest, S., Weimer, W.: A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. In: 34th In- ternational Conference on Software Engineering (ICSE). pp. 3–13 (2012).https: //doi.org/10.1109/ICSE.2012.6227211

    2012

  20. [29]

    In: Proceedings of the 28th ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis

    Liu, K., Koyuncu, A., Kim, D., Bissyandé, T.F.: Tbar: revisiting template-based automated program repair. In: Proceedings of the 28th ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis. p. 31–42. ISSTA 2019, Asso- ciation for Computing Machinery, New York, ...

    2019

  21. [30]

    In: 38th International Conference on Software Engineering (ICSE)

    Mechtaev, S., Yi, J., Roychoudhury, A.: Angelix: scalable multiline program patch synthesis via symbolic analysis. In: 38th International Conference on Software Engineering (ICSE). pp. 691–701 (2016). https://doi.org/10.1145/2884781. 2884807

    2016 doi

  22. [31]

    In: 35th International Conference on Software Engineering (ICSE)

    Nguyen, H.D.T., Qi, D., Roychoudhury, A., Chandra, S.: Semfix: program repair via semantic analysis. In: 35th International Conference on Software Engineering (ICSE). pp. 772–781 (2013).https://doi.org/10.1109/ICSE.2013.6606623

    2013

  23. [32]

    In: Proceedings of the 21st In- ternational Conference on Mining Software Repositories

    Ni, C., Shen, L., Yang, X., Zhu, Y., Wang, S.: Megavul: A c/c++ vulnerability dataset with comprehensive code representations. In: Proceedings of the 21st In- ternational Conference on Mining Software Repositories. MSR ’24, Association for Computing Machinery, New York, NY, US...

    2024

  24. [33]

    Ogata, M., Franklin, J., Voas, J., Sritapan, V., Quirolgico, S.: Vetting the security of mobile applications. Tech. rep., National Institute of Standards and Technology (2019). https://doi.org/10.6028/NIST.SP.800-163r1

    2019 doi

  25. [34]

    IEEE Trans

    Okutan, A., Mell, P., Mirakhorli, M., Khokhlov, I., Santos, J.C.S., Gonzalez, D., Simmons, S.: Empirical validation of automated vulnerability curation and char- acterization. IEEE Trans. Software Eng.49(5), 3241–3260 (2023). https://doi. org/10.1109/TSE.2023.3250479, https://...

    2023

  26. [35]

    In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

    2002

  27. [36]

    Piskachev, G., Becker, M., Bodden, E.: Can the configuration of static analyses make resolving security vulnerabilities more effective? - A user study. Empir. Softw. Eng. 28(5), 118 (2023).https://doi.org/10.1007/S10664-023-10354-3, https: //doi.org/10.1007/s10664-023-10354-3

    2023 doi

  28. [37]

    arXiv preprint arXiv:2009.10297 (2020)

    Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., Ma, S.: Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  29. [38]

    In: 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024

    Risse, N., Böhme, M.: Uncovering the limits of machine learning for automatic vulnerability detection. In: 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association (2024), https://www.usenix.org/conference/usenixsecuri...

    2024

  30. [39]

    arXiv preprint arXiv:2109.00859 (2021)

    Wang, Y., Wang, W., Joty, S., Hoi, S.C.: Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  31. [40]

    In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Soft- ware Testing and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023

    Wu, Y., Jiang, N., Pham, H.V., Lutellier, T., Davis, J., Tan, L., Babkin, P., Shah, S.: How effective are neural networks for fixing security vulnerabilities. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Soft- ware Testing and Analysis, ISSTA 2023, Seattl...

    2023

  32. [41]

    IEEE Transactions on Software Engineering 43, 34–55 (2017)

    Xuan, J., Martinez, M., DeMarco, F., Clément, M., Marcote, S.L., Durieux, T., Le Berre, D., Monperrus, M.: Nopol: automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering 43, 34–55 (2017). https://doi.org/10.1109/TSE.2016.2560811

    2017

  33. [42]

    IEEE Access 8, 166335–166346 (2020)

    Zhao, R., Wang, J., Zheng, X., Wen, J., Rao, L., Zhao, J.: Maritime visible image classification based on double transfer method. IEEE Access 8, 166335–166346 (2020)

    2020

  34. [60]

    https://doi.org/10.1007/978-3-319-10374-7_3 , https: //doi.org/10.1007/978-3-319-10374-7_3

    Springer (2014). https://doi.org/10.1007/978-3-319-10374-7_3 , https: //doi.org/10.1007/978-3-319-10374-7_3

    2014 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.