Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Fine-tuned vulnerability-repair models are overestimated because they are evaluated on overlapping data and often memorize patches rather than learn to fix unseen bugs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:49 UTC pith:FKHS3LB7

load-bearing objection Strong leakage finding buried under an internally inconsistent test-based benchmark; worth peer review after the RQ4 headline counts are reconciled with the paper's own manual labels. the 4 major comments →

arxiv 2512.22633 v2 pith:FKHS3LB7 submitted 2025-12-27 cs.SE

Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair

Woorim Han , Yeongjun Kwak , Miseon Yu , Kyeongmin Kim , Younghan Lee , Hyungon Moon , Yunheung Paek This is my paper
classification cs.SE
keywords automated vulnerability repairfine-tuned language modelstrain-test overlapdata leakagechronological splitexact matchtest-based evaluationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that state-of-the-art fine-tuned automated vulnerability repair (AVR) models do not generalize to unseen vulnerabilities, and that standard evaluation settings hide this. Three lines of evidence support the claim: semantic-preserving transformations cause large score drops, indicating reliance on spurious surface features; a chronological re-split that removes overlapping commit-level data drops exact-match scores from roughly 18-20% to about 5%; and a new test-based benchmark, L-AVRBench, shows that token-overlap metrics (EM, BLEU, CodeBLEU) cannot tell a real fix from a broken patch. The paper introduces L-AVRBench, a 70-vulnerability test suite with security proofs-of-concept and unit tests, and reports that state-of-the-art models repair only 4, 15, and 5 vulnerability IDs respectively. The findings imply that published capability numbers for fine-tuned AVR models should be re-examined and that evaluation should move beyond match-based metrics toward test-based validation.

Core claim

State-of-the-art fine-tuned AVR models are systematically overestimated: a chronological re-split that removes commit-duplicated overlap drops VulRepair's exact-match score from 18.30% to 4.96% and VulMaster's from 20.47% to 5.15%, with over 85% of exact-match successes traced to a single overlapping commit. Match-based metrics (EM, BLEU, CodeBLEU) cannot distinguish functionally correct from failing patches. Using the new test-based benchmark L-AVRBench (70 real vulnerabilities, security PoCs plus unit tests), VulRepair, VulMaster, and VulAdvisor repair 4, 15, and 5 vulnerability IDs respectively, with only one exact match.

What carries the argument

Three probes carry the argument: semantic-preserving transformations (renaming identifiers, reordering statements, inserting inert code) that expose sensitivity to surface syntax; a chronological re-split of the data so no commit-duplicated patch appears in more than one partition, which isolates data leakage; and L-AVRBench, a test-based benchmark that combines security proof-of-concept triggers and unit tests to judge whether a generated patch actually eliminates the vulnerability while preserving behavior. The commit-overlap analysis is the central mechanism: patches from a single Git commit are near-identical, so random splits leak duplicates into training and test sets, inflating all to

Load-bearing premise

The benchmark's automated pass on one proof-of-concept and unit tests is assumed to mean a true repair; the paper's own manual inspection found only 6 of 24 automatically 'passed' vulnerability IDs to be genuinely reasonable patches, so the reported repair counts (15, 5, 4) rest on test oracles that the paper itself shows to be lenient.

What would settle it

Have independent security engineers write additional proofs-of-concept for the same underlying bug for each L-AVRBench ID and re-run the 131 patches that passed the original functionality and security tests; if a large fraction still pass the new PoCs and are judged reasonable, the claim that these models rarely repair unseen vulnerabilities is weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Published EM, BLEU, and CodeBLEU scores for fine-tuned AVR models overstate the ability to repair unseen vulnerabilities; future leaderboards should adopt non-overlapping, chronologically split data.
  • Match-based metrics alone are not enough; test-based evaluation changes which model appears strongest and reveals trade-offs between security and functionality.
  • The L-AVRBench infrastructure gives the community a reusable way to assess patch correctness and security beyond token matching.
  • Researchers should treat near-duplicate samples from a single commit as a form of data leakage and report results under both random and chronological splits.
  • If training on truly non-overlapping data, models may improve in real-world generalization, since reported high scores were partly memorization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The chronological-split lesson generalizes beyond AVR: any code-generation or program-repair dataset mined from git commits where one commit patches multiple functions is exposed to the same overlap inflation, so similar re-splitting may change reported results in those tasks too.
  • A test-based benchmark can act as a selection signal in practice: generating many patch candidates and filtering them by security PoCs and unit tests turns a weak one-shot repair model into a viable semi-automated repair workflow, a direction the paper's 36-attempt evaluation implicitly supports but does not itself implement.
  • L-AVRBench currently skews toward memory-safety and crash-triggering vulnerabilities; extending it with logic, concurrency, and non-crash security bugs could alter the relative ranking of models that use AST/CWE hints versus plain fine-tuning.
  • The paper's manual audit of the 24 IDs that passed its automated tests found only 6 truly reasonable patches, so even the test-based counts likely overstate real repair capability; a stricter human-verified accounting would put the success rates lower.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper reproduces three fine-tuned AVR models (VulRepair, VulMaster, VulAdvisor) and investigates whether their reported repair performance reflects genuine generalization. It presents three lines of evidence: (RQ1) EM/BLEU/CodeBLEU drop when semantically equivalent code transformations are applied; (RQ2) random data splitting causes same-commit overlap, and a chronological re-split reduces VulRepair EM from 18.30% to 4.96% and VulMaster EM from 20.47% to 5.15%; (RQ3) a new test-based benchmark, L-AVRBench, is used to argue that EM/BLEU/CodeBLEU do not correlate with functional correctness and security; and (RQ4) test-based evaluation yields headline repair counts of 15, 5, and 4 vulnerability IDs for VulMaster, VulAdvisor, and VulRepair, respectively. The paper also reports a manual review of the 131 patches that passed the automated tests, finding that only 6 of 24 IDs are 'Reasonable' repairs.

Significance. If the RQ2 result is correct, it is a substantial empirical contribution: the same-commit overlap rates of 85.76% and 88.18% for correctly repaired EM=1 samples, together with the chronological re-split drop, imply that previously published EM scores for VulRepair and VulMaster substantially overstate their ability to repair unseen vulnerabilities. The use of externally sourced vulnerability data (ExtractFix, Magma, public CVEs) and the direct commit-overlap measurement make this finding credible and important. The L-AVRBench resource is also a useful step toward execution-based evaluation. However, the test-based claims in RQ3 and RQ4 are not currently supported because the paper's own manual analysis shows that the automated oracle labels are mostly false positives; the correlation analysis lacks statistical measures; and the evaluations are performed on retrained, context-emitting variants rather than the original published models. These issues must be addressed before the benchmark-based capability conclusions can be accepted.

major comments (4)
  1. [§5.2/§5.3, Tables 7 and 8; abstract] The central RQ4 result (VulMaster 15 IDs, VulAdvisor 5 IDs, VulRepair 4 IDs) is based on the automated L-AVRBench verdicts in Table 7. However, §5.3 manually reviews all 131 patches that passed both functionality and security tests and classifies only 6 of 24 vulnerability IDs as 'Reasonable': VulMaster 4, VulRepair 2, VulAdvisor 0. Thus, under the paper's own gold standard, 75% of the automated positive IDs are not valid repairs, and VulAdvisor's 5 reported successes are all classified as NF or NR. Table 7 counts an ID if any of 36 generated patches passes, while Table 8 reports only the best patch per ID, so the automated numbers are additionally optimistic. Because §5.3 itself acknowledges that functionality tests are insufficient (citing GenProg and Pearce et al.), the automated oracle cannot bear the weight of the RQ4 capability claims. The paper should make manual classification pr
  2. [§4.1, Figure 4] The claim that BLEU and CodeBLEU 'exhibit weak correlations' with functional correctness and security is supported only by visual overlap of density plots. No rank correlation, AUC, effect size, or confidence interval is reported, and with 7,560 generated patches but only 131 positive labels, quantification is essential. More fundamentally, the pass/fail labels used in Figure 4 are the same L-AVRBench oracle labels that §5.3 shows to be largely false positives. A correlation computed against a noisy, mostly wrong oracle does not measure correlation with true repair quality. Rerun these analyses with corrected labels or a validated oracle and report statistical measures.
  3. [§5.1 and §2.2] For the L-AVRBench evaluation, the models are not the published SOTA models but retrained variants: all three models are retrained to emit diffs with before/after context, line-based formatting is restored, and 10–15 additional epochs are used. The paper reports EM changes (e.g., VulRepair 18.30 to 16.32) and then attributes RQ3/RQ4 outcomes to the original systems. The 'impact is not significant' claim is based on point estimates without variance or significance testing. Please provide a stronger equivalence argument, evaluate the original models with an application procedure that handles their exact output format, or explicitly frame RQ3/RQ4 as characterizing context-emitting variants rather than the published models.
  4. [§3.2, Table 4] The chronological re-split is presented as an estimate of the impact of overlapping data, but it changes two variables simultaneously: it removes same-commit overlap and it shifts the test distribution to later vulnerabilities, which may differ in project, type, and difficulty. Keeping subset sizes equal does not control for composition. The direct same-commit overlap measurement (Table 3) is strong evidence of leakage, but the magnitude of the EM drop (18.30 to 4.96 and 20.47 to 5.15) conflates overlap removal with temporal distribution shift. A random split that groups all samples from the same commit into one partition would isolate the overlap effect from the time shift and should be added or discussed as a threat to the quantitative interpretation.
minor comments (4)
  1. [Table 4] The VulMaster BLEU drop is reported as -15.32, but 59.81 - 50.62 = 9.19. The EM drop (-15.32) appears to have been copied into the BLEU row.
  2. [§5.3, references [3, 36]] The text says 'Qi et al. [36] showed that despite GenProg claiming to fix 55 bugs [3]...' Reference [36] is the Le Goues et al. GenProg paper, not Qi et al., and reference [3] is Austin et al. 2021, not the GenProg bug-count claim. These citations should be corrected.
  3. [§3.1, Table 2] The transformation labels skip t7 and t9. The text says comment removal and whitespace insertion were excluded, but does not map those exclusions to the missing labels. Please state explicitly which transformations correspond to t7 and t9.
  4. [§2.2] The VulAdvisor evaluation uses Ground Truth suggestions rather than VulAdvisor's own suggestion model. This makes the evaluation an oracle-aided upper bound rather than an end-to-end assessment of the published VulAdvisor system. This caveat should appear prominently in the experimental setup and in the conclusions, since it affects the cross-model comparison in RQ4.

Circularity Check

0 steps flagged

No circular derivation: empirical claims rest on external benchmarks and independent re-splits; L-AVRBench oracle validity is a correctness risk, not circularity.

full rationale

The paper's central claims are empirical and externally anchored: RQ1 uses semantic-preserving transformations from prior work [56] applied to test sets; RQ2 uses a chronological re-split of externally sourced BigVul/CVEFixes data to measure overlap-driven inflation; RQ3 and RQ4 use L-AVRBench, assembled from ExtractFix, Magma, and public CVE reports, with same-CVE samples removed from training before evaluation. No equation or claimed prediction is fitted to, or defined in terms of, the quantity it is supposed to predict. There are no self-citations by the present authors; the cited AVR models are external prior work with independently published scores. The paper even includes a manual analysis (§5.3, Table 8) that tempers the automated test-based counts and explicitly acknowledges the PoC-only limitation: 'Our evaluation relies on PoCs to validate repairs, which only confirm that a particular failure is addressed and do not guarantee that all instances of the vulnerability are eliminated.' The most concerning issue—that L-AVRBench's automated verdicts are used as ground truth for both the metric-correlation analysis and RQ4 capability counts while the paper's own manual review finds only 6 of 24 passed IDs 'Reasonable'—is an internal-validity and oracle-correctness threat, not a circular reduction. It should be weighed under correctness risk, but it does not make the derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims rest on three domain assumptions: transformations preserve semantics, chronological splitting removes leakage, and PoC/unit tests measure repair quality. The last is partially contradicted by the authors' manual analysis. No free fitted parameters or invented explanatory entities; L-AVRBench is an unreleased dataset/benchmark artifact, not a postulated mechanism.

axioms (4)
  • domain assumption Semantic-preserving transformations preserve functionality in the AVR test set.
    RQ1 attributes EM drops to spurious-feature reliance; if a transformation changed semantics, the drop would not indicate overfitting. Adopted from Risse & Böhme (vulnerability detection) and asserted verified without released reproduction.
  • domain assumption Chronological commit-based splitting eliminates train/test overlap sufficient to measure true generalization.
    RQ2 assumes older-data training vs newer-data test removes leakage; it catches same-commit duplicates but not cross-commit near-duplicates, and confounds leakage removal with distribution shift over time.
  • domain assumption PoC security tests plus project unit tests determine whether a patch is a correct repair.
    L-AVRBench bases Functionality/Security verdicts on these tests; the paper's own manual analysis (§5.3) later shows only 6/24 such verdicts are 'Reasonable', so this premise is load-bearing and partially contradicted.
  • ad hoc to paper Retrained context-emitting variants faithfully represent the published SOTA models.
    To apply patches, models were retrained to produce context diffs with newlines; EM drops (e.g., VM 20.47→18.6) show the evaluated models differ from the published ones.

pith-pipeline@v1.3.0-alltime-deepseek · 222 in / 10451 out tokens · 162975 ms · 2026-08-03T13:49:05.159796+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair." pith.science (2026). https://pith.science/paper/FKHS3LB7

@misc{pith2026251222633,
  author       = {Pith},
  title        = {Pith review of: Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKHS3LB7}},
  note         = {Machine review of arXiv:2512.22633}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Learning-based automated vulnerability repair (AVR) techniques that utilize fine-tuned language models have shown promise in generating vulnerability patches. However, questions remain about their ability to repair unseen vulnerabilities. Our empirical study reveals that state-of-the-art models often overfit to the training set and are evaluated using training, validation, and test sets that are not mutually exclusive. Furthermore, relying on match-based metrics that compare generated patches to reference fixes at the token level has some limitations, failing to account for the possibility of various valid ways to patch the vulnerability. In this paper, we examine the capabilities of state-of-the-art fine-tuned AVR models and the adequacy of match-based evaluation metrics in three ways. First, we apply semantic-preserving transformations to test sets in order to determine whether models truly learn robust vulnerability-repair patterns or simply rely on spurious features. Second, we re-split the training, validation, and test sets to be mutually exclusive and evaluate the models on the revised test set to assess their generalization capabilities. Third, we introduce L-AVRBench, a test-based benchmark tailored for learning-based AVR, to overcome the limitations of match-based metrics and examine the AVR models' true repair capabilities.

Figures

Figures reproduced from arXiv: 2512.22633 by Hyungon Moon, KyeongMin Kim, Miseon Yu, Woorim Han, Yeongjun Kwak, Younghan Lee, Yunheung Paek.

Figure 1
Figure 1. Figure 1: Learning curves for SOTA Fine-tuned language mod [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (CVE-2015-8631) Overlapping data caused by ran [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (CVE-2016-9560) Vulnerable code that requires an [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The probability densities of existing metrics for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (LB4) Correctly repaired example when the refer [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution

    cs.SE 2026-05 unverdicted novelty 7.0

    EvoRepair is the first experience-based self-evolving agent framework for automated vulnerability repair, reporting 90.46% overall success on PATCHEVAL and SEC-bench benchmarks.

Reference graph

Works this paper leans on

71 extracted references · 13 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Uni- fied Pre-training for Program Understanding and Generation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies. 2655–2668

    2021

  2. [2]

    Leonhard Applis, Annibale Panichella, and Arie van Deursen. 2021. Assessing robustness of ml-based program analysis tools using metamorphic program transformations. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1377–1381

    2021

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  4. [4]

    Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. InProceed- ings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. 30–39

    2021

  5. [5]

    Pavol Bielik and Martin Vechev. 2020. Adversarial robustness for code. InInter- national Conference on Machine Learning. PMLR, 896–907

    2020

  6. [6]

    Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation.arXiv preprint arXiv:2202.07654(2022)

    Pith/arXiv arXiv 2022

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  8. [8]

    Zimin Chen, Steve Kommrusch, and Martin Monperrus. 2022. Neural transfer learning for repairing security vulnerabilities in c code.IEEE Transactions on Software Engineering49, 1 (2022), 147–165

    2022

  9. [9]

    CWE Community. 2023. Common Weakness Enumeration: CWE. https://cwe. mitre.org/ Accessed: March 2025

    2023

  10. [10]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We?arXiv preprint arXiv:2403.18624(2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2023. Code- Score: Evaluating Code Generation by Learning Code Execution.arXiv preprint arXiv:2301.09043(2023)

    Pith/arXiv arXiv 2023

  12. [12]

    Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: Quantifying data leakage in NLP performance evaluation.arXiv preprint arXiv:2102.01818(2021)

    Pith/arXiv arXiv 2021

  13. [13]

    Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software203 (2023), 111741

    2023

  14. [14]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. InProceedings of the 17th International Conference on Mining Software Repositories(Seoul, Republic of Korea)(MSR ’20). Association for Computing Machinery, New York, NY, USA, 508–512. doi:10.1145/3379597.3387501

    arXiv 2020

  15. [15]

    Forbes Technology Council. 2023. Zero-Day Vulnerabilities: 17 Consequences and Complications. https://www.forbes.com/sites/forbestechcouncil/2023/05/ 26/zero-day-vulnerabilities-17-consequences-and-complications Accessed on: Jan, 2025

    2023

  16. [16]

    Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung

  17. [17]

    2024.2023 CVE Data Review

    Jerry Gamblin. 2024.2023 CVE Data Review. https://jerrygamblin.com/2024/01/ 03/2023-cve-data-review/ Accessed: 2025-02-24

    arXiv 2024

  18. [18]

    Xiang Gao, Sergey Mechtaev, and Abhik Roychoudhury. 2019. Crash-avoiding program repair. InProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 8–18

    2019

  19. [19]

    Xiang Gao, Bo Wang, Gregory J Duck, Ruyi Ji, Yingfei Xiong, and Abhik Roy- choudhury. 2021. Beyond tests: Program vulnerability repair via crash constraint extraction.ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 2 (2021), 1–27

    2021

  20. [20]

    Kyle Gorman and Steven Bedrick. 2019. We Need to Talk About Standard Splits. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 5360–5364. doi:10. 18653/v1/P19-1530

    2019

  21. [21]

    Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2022. FixEval: Execution-based Evaluation of Program Fixes for Program- ming Problems.arXiv preprint arXiv:2206.07796(2022)

    Pith/arXiv arXiv 2022

  22. [22]

    Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. 2020. Magma: A ground- truth fuzzing benchmark.Proceedings of the ACM on Measurement and Analysis of Computing Systems4, 3 (2020), 1–29

    2020

  23. [23]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Mea- suring coding challenge competence with apps.arXiv preprint arXiv:2105.09938 (2021)

    Pith/arXiv arXiv 2021

  24. [24]

    Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu

    Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu

  25. [25]

    Seongjoon Hong, Junhee Lee, Jeongsoo Lee, and Hakjoo Oh. 2020. SAVER: scalable, precise, and safe memory-error repair. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 271–283

    2020

  26. [26]

    Heqing Huang, Peisen Yao, Hung-Chun Chiu, Yiyuan Guo, and Charles Zhang

  27. [27]

    Zhen Huang, Mariana DAngelo, Dhaval Miyani, and David Lie. 2016. Talos: Neutralizing vulnerabilities with security workarounds for rapid response. In 2016 IEEE Symposium on Security and Privacy (SP). IEEE, 618–635

    2016

  28. [28]

    Zhen Huang, David Lie, Gang Tan, and Trent Jaeger. 2019. Using safety properties to generate vulnerability patches. In2019 IEEE symposium on security and privacy (SP). IEEE, 539–554

    2019

  29. [29]

    Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering.arXiv preprint arXiv:2007.01282(2020)

    Pith/arXiv arXiv 2020

  30. [30]

    Tiantian Ji, Yue Wu, Chang Wang, Xi Zhang, and Zhongru Wang. 2018. The coming era of alphahacking?: A survey of automatic software vulnerability detection, exploitation and patching techniques. In2018 IEEE third international conference on data science in cyberspace (DSC). IEEE, 53–60

    2018

  31. [31]

    Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. 2018. Predicting the generalization gap in deep networks with margin distributions.arXiv preprint arXiv:1810.00113(2018)

    Pith/arXiv arXiv 2018

  32. [32]

    Rie Johnson and Tong Zhang. 2023. Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training. InProceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)

    2023

  33. [33]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of ex- isting faults to enable controlled testing studies for Java programs. InProceedings of the 2014 international symposium on software testing and analysis. 437–440

    2014

  34. [34]

    Brian Kishiyama, Young Lee, and Jeong Yang. 2024. Improving VulRepair’s Perfect Prediction by Leveraging the LION Optimizer.Applied Sciences14, 13 (2024), 5750

    2024

  35. [35]

    Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Langauge Technology (HLT-NAACL 2003). Association for Computational Linguistics, 48–54

    2003

  36. [36]

    Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2011. Genprog: A generic method for automatic software repair.Ieee transactions on software engineering38, 1 (2011), 54–72

    2011

  37. [37]

    Junhee Lee, Seongjoon Hong, and Hakjoo Oh. 2018. Memfix: static analysis-based repair of memory deallocation errors for c. InProceedings of the 2018 26th ACM Joint meeting on European software engineering conference and symposium on the foundations of software engineering. 95–106

    2018

  38. [38]

    Frank Li and Vern Paxson. 2017. A large-scale empirical study of security patches. InProceedings of the 2017 ACM SIGSAC Conference on Computer and Communica- tions Security. 2201–2215

    2017

  39. [39]

    Yuan Li, Wende Tan, Zhizheng Lv, Songtao Yang, Mathias Payer, Ying Liu, and Chao Zhang. 2022. Pacmem: Enforcing spatial and temporal memory safety via arm pointer authentication. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1901–1915

    2022

  40. [40]

    Pei Liu, Bo Lin, Yihao Qin, Cheng Weng, and Liqian Chen. 2024. T-RAP: A Template-guided Retrieval-Augmented Vulnerability Patch Generation Approach. InProceedings of the 15th Asia-Pacific Symposium on Internetware. 105–114

    2024

  41. [41]

    Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. 2024. Exploring{ChatGPT’s} Capabilities on Vulnerability Management. In33rd USENIX Security Symposium (USENIX Security 24). 811–828

    2024

  42. [42]

    Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, and Sanjiv Kumar. 2024. Regression-aware Inference with LLMs. arXiv:2403.04182 [cs.CL] https://arxiv.org/abs/2403.04182

    Pith/arXiv arXiv 2024

  43. [43]

    Siqi Ma, David Lo, Teng Li, and Robert H Deng. 2016. Cdrep: Automatic repair of cryptographic misuses in android applications. InProceedings of the 11th ACM on Asia conference on computer and communications security. 711–722

    2016

  44. [44]

    Antonio Mastropaolo, Vittoria Nardone, Gabriele Bavota, and Massimiliano Di Penta. 2024. How the Training Procedure Impacts the Performance of Deep Learning-based Vulnerability Patching. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 150–159

    2024

  45. [45]

    National Institute of Standards and Technology. [n. d.]. National Vulnerability Database. https://nvd.nist.gov/ Accessed: 2025-07-15

    2025

  46. [46]

    Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shaohua Wang. 2024. MegaVul: A C/C++ vulnerability dataset with comprehensive code representations. In Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair ICSE ’26, April 12–18, 2026, Rio de Janeiro, Brazil Proceedings of the 21st International Conference on Mining Softwa...

    2024

  47. [47]

    2022.extractfix-benchmark

    nus-apr. 2022.extractfix-benchmark. https://github.com/nus-apr/extractfix- benchmark Accessed: 2025-02-24

    2022

  48. [48]

    Juri Opitz, Carolina Scarton, and Lucia Specia. 2023. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 1234–1245

    2023

  49. [49]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  50. [50]

    Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2339–2356

    2023

  51. [51]

    Marek Polacek. 2014. GCC undefined behavior sanitizer-UBSan.GCC undefined behavior sanitizer-UBSan(2014)

    2014

  52. [52]

    Md Rafiqul Islam Rabin, Nghi DQ Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2021. On the generalizability of neural program mod- els with respect to semantic-preserving program transformations.Information and Software Technology135 (2021), 106552

    2021

  53. [53]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. InProceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2383–2392

    2016

  54. [54]

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet Classifiers Generalize to ImageNet?. InProceedings of the 36th International Conference on Machine Learning (ICML). PMLR, 5389–5400. http: //proceedings.mlr.press/v97/recht19a.html

    2019

  55. [55]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020)

    Pith/arXiv arXiv 2020

  56. [56]

    Niklas Risse and Marcel Böhme. 2024. Uncovering the Limits of Machine Learn- ing for Automatic Vulnerability Detection. In33rd USENIX Security Symposium (USENIX Security 24). 4247–4264

    2024

  57. [57]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage?Advances in Neural Information Processing Systems36 (2023), 55565–55581

    2023

  58. [58]

    Kostya Serebryany. 2017. OSS-Fuzz - Google’s continuous fuzzing service for open source software. USENIX Association, Vancouver, BC

    2017

  59. [59]

    2012.{AddressSanitizer}: A fast address sanity checker

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012.{AddressSanitizer}: A fast address sanity checker. In2012 USENIX annual technical conference (USENIX ATC 12). 309–318

    2012

  60. [60]

    Konstantin Serebryany and Timur Iskhodzhanov. 2009. ThreadSanitizer: data race detection in practice. InProceedings of the Workshop on Binary Instrumentation and Applications(New York, New York, USA)(WBIA ’09). Association for Computing Machinery, New York, NY, USA, 62–71. doi:10.1145/1791194.1791203

    arXiv 2009

  61. [61]

    Yonghee Shin, Andrew Meneely, Laurie Williams, and Jason A Osborne. 2010. Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities.IEEE transactions on software engineering37, 6 (2010), 772–787

    2010

  62. [62]

    Statista. 2024. Number of reported Common Vulnerabilities and Exposures (CVEs) worldwide from 1999 to 2024. https://www.statista.com/statistics/500755/ worldwide-common-vulnerabilities-and-exposures/ Accessed: 2025-02-24

    2024

  63. [63]

    Evgeniy Stepanov and Konstantin Serebryany. 2015. MemorySanitizer: Fast detector of uninitialized memory use in C++. In2015 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 46–55. doi:10.1109/CGO. 2015.7054186

    arXiv 2015

  64. [64]

    Rijnard van Tonder and Claire Le Goues. 2018. Static automated program repair for heap properties. InProceedings of the 40th International Conference on Software Engineering. 151–162

    2018

  65. [65]

    Weishi Wang, Yue Wang, Shafiq Joty, and Steven CH Hoi. 2023. Rap-gen: Retrieval- augmented patch generation with codet5 for automatic program repair. InPro- ceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 146–158

    2023

  66. [66]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. arXiv:2109.00859 [cs.CL] https://arxiv.org/abs/2109. 00859

    Pith/arXiv arXiv 2021

  67. [67]

    Jian Zhang, Chong Wang, Anran Li, Wenhan Wang, Tianlin Li, and Yang Liu. 2024. VulAdvisor: Natural Language Suggestion Generation for Software Vulnerability Repair. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1932–1944

    2024

  68. [68]

    Xin Zhou, Kisub Kim, Bowen Xu, Donggyun Han, and David Lo. 2024. Out of Sight, Out of Mind: Better Automatic Vulnerability Repair by Broadening Input Ranges and Sources. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 88, 13 pag...

    arXiv 2024

  69. [2012]

    In2012 34th International Conference on Software Engineering (ICSE)

    On the naturalness of software. In2012 34th International Conference on Software Engineering (ICSE). 837–847. doi:10.1109/ICSE.2012.6227135

    arXiv 2012

  70. [2022]

    InPro- ceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Singapore, Singapore) (ESEC/FSE 2022)

    VulRepair: a T5-based automated software vulnerability repair. InPro- ceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 935–947. doi:10.1145/3540250.3549098

    arXiv 2022

  71. [2024]

    In2024 IEEE Sympo- sium on Security and Privacy (SP)

    Titan: efficient multi-target directed greybox fuzzing. In2024 IEEE Sympo- sium on Security and Privacy (SP). IEEE, 1849–1864