Pith. sign in

REVIEW 4 major objections 8 minor 30 references

Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generation

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Showing GPT-4 both the vulnerable and the patched code yields useful unit-test templates in 68.5% of cases.

desk verdict New prompt design (vulnerable+patched code) yields a credible 7.5% fully automatic hit rate; the 68.5% 'useful' claim is fragile as stated but worth a revision. read the letter →

arxiv 2506.11559 v1 pith:ACPNCOC6 submitted 2025-06-13 cs.SE cs.AI

Gábor Antal , Dénes Bán , Martin Isztin , Rudolf Ferenc , Péter Hegedűs This is my paper
classification cs.SEcs.AI
keywords unittestgenerationvulnerability-witnessingtestslargelanguagemodelsGPT-4JavavulnerabilitiesCWEfocalcontextautomatedtesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using 50 real Java vulnerabilities whose fixes touch a single class, this paper asks whether GPT-4 can write a unit test that fails on the vulnerable code and passes on the patched code. The model compiled 66.5% of its generated tests without any fine-tuning, but strict automatic validation—the test must fail before the patch and pass after it—succeeded in only 7.5% of cases. A manual review by two developers found 68.5% of the tests useful as templates that could be turned into working vulnerability-witnessing tests with minor edits. The authors conclude that large language models are not yet ready to run fully autonomously for this task, but they can already assist developers and help researchers annotate untested vulnerabilities.

What carries the argument

The load-bearing mechanism is the before/after code pair combined with four levels of focal context, called L0 through L3, that grow from the vulnerable method alone to the full class skeleton. The prompt first shows the vulnerable method with its class context, then the patched method, and asks the model to act as a senior software tester; a closing emotional appeal asks for the model's best effort. When automatic execution fails, the pipeline re-prompts GPT-4 with the compiler or test log so the model can correct its own output. Each generated test is then judged by the vulnerability-witness criterion: fail on the original code, pass on the patched code.

What would settle it

Run the identical prompt pipeline on a random sample of vulnerabilities without the one-class filter and check whether the syntactic, semantic, and usefulness rates stay near 66.5%, 7.5%, and 68.5%; a sizable drop would show the headline figures are an artifact of the easy subset.

Watch

Extended reading notes

Core claim

The paper's central claim is that presenting GPT-4 with the vulnerable and fixed versions of a method, embedded in a focal code context, yields a vulnerability-witnessing test often enough to be practically useful. The headline numbers are 66.5% syntactic correctness, 7.5% semantic correctness under the strict fail-before/pass-after criterion, and 68.5% subjective usefulness. The gap between 7.5% and 68.5% is the key insight: most generated tests fail only because of missing imports, wrong call signatures, or small scaffolding mistakes, not because the model misunderstood the vulnerability. With the before/after code pair as evidence, the model usually points at the right method and the right malicious input, leaving a human to repair the mechanics.

Load-bearing premise

The results rest on the assumption that the 50 fixes that touch just one class (usually one method) represent real-world vulnerabilities broadly enough; if that filter also selects simpler cases, the reported percentages will not carry over to the full population of vulnerabilities.

Editorial extensions

If this is right

  • A developer fixing a known vulnerability can expect GPT-4 to supply a compiling test skeleton about two times out of three, and a useful starting template about 68.5% of the time.
  • The fail-before/pass-after criterion gives an automatic, objective signal for evaluating LLM-generated security tests, so the same pipeline can be reused as a benchmark.
  • Researchers can apply the method to annotate vulnerabilities that currently lack tests, expanding the pool of labeled examples for training or evaluating repair models.
  • Because adding CWE identifiers and extra context did not consistently help, prompt construction—not just model capacity—is a major lever on generation quality.
  • Fully automated adoption is not yet justified; the 7.5% strict success rate means a human review step remains necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-class filter probably makes the task easier than the general vulnerability population; a random sample of multi-file fixes would likely show lower usefulness rates, since changes spanning several classes strain the model's context and reasoning.
  • The 7.5% strict rate may understate the ceiling: fixing the two most common failure modes (missing imports and wrong call signatures) with a post-processing pass could lift the fully automatic success rate well above 7.5% before any fine-tuning.
  • A direct comparison against a prompt showing only the patched method would isolate how much the vulnerable version contributes; if templates degrade without it, the before/after pair is the actual driver of usefulness.
  • A time-to-fix or edit-distance metric would turn the subjective 68.5% figure into a quantitative benchmark, making the usefulness claim reproducible across evaluators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper investigates whether GPT-4 Turbo can generate vulnerability-witnessing unit tests for real Java vulnerabilities from the VUL4J dataset. For a manually selected subset of 50 vulnerabilities whose fixes affect a single class and mostly a single method, the authors construct four focal-context levels (L0-L3), prompt the model with the vulnerable and fixed code, and run up to five feedback-based refinement iterations. Generated tests are executed in the VUL4J Docker environment against both the vulnerable and fixed versions, yielding an automatic syntactic-correctness rate of 66.5% and an automatic fail-before/pass-after semantic-correctness rate of 7.5% across 200 context runs. A subjective manual review by two developers, one of whom is an author, labels 68.5% of the generated tests as useful templates requiring minimal human refinement. The paper also reports context-level effects, a prompt ablation, CWE-group trends, and several qualitative observations about feedback-based generation.

Significance. If the claims hold, this is a useful empirical contribution to the emerging area of LLM-based test generation: it uses real vulnerabilities and fixes, validates results by actually executing generated tests in the original project environments, and provides an artifact (Zenodo) plus a prompt-ablation analysis. The automatic PASS/FAIL evaluation is externally grounded and therefore gives a credible lower bound on fully automated capability (7.5%). The higher subjective-usefulness claim, however, is the central basis for the paper's conclusion that GPT-4 'can already provide helpful assistance,' and that claim currently rests on an unvalidated subjective rating procedure. The paper is also transparent about several limitations (single model, no fine-tuning, limited prompt variants, non-determinism), which is commendable. Overall, the study is a reasonable initial evidence point, but the strength of its headline claims exceeds what the current evaluation procedures support.

major comments (4)
  1. [Section 4.3 and Section 3.3] The 68.5% usefulness rate is the load-bearing evidence for the paper's central conclusion, but the manual evaluation that produces it is not validated. Section 3.3 states that two developers labeled results and that labels were assigned when both agreed, but no inter-rater reliability statistic (e.g., Cohen's kappa) is reported, no information is given about how disagreements were resolved, the evaluation is not blinded, and one of the two raters is an author. The criterion 'minimal human refinement' is never quantified; Table 1 contains many Manual=OK entries where the automatic result is ERR-ERR or FAIL-FAIL (e.g., VUL4J-04, VUL4J-18, VUL4J-52 at all levels), so the usefulness label is not anchored to any observable behavioral threshold. Without a quantitative effort measure (e.g., number of edits, edit distance, or time to repair), and without a blinded second rater or an agreement measure, the claim that 68.5% of tests are useful cannot be distinguished from an optimistic holistic impression. This needs to be fixed before the main conclusion can be accepted.
  2. [Section 3.1 and Section 7] The 50-subject subset is not a random sample of VUL4J: selection is restricted to fixes affecting a single class and, in most cases, a single method, chosen to fit the model's context window. The paper does not compare this subset with the full VUL4J population on characteristics that plausibly affect difficulty, such as patch size, number of changed methods, CWE distribution, or project age. Because the headline percentages (66.5%, 7.5%, 68.5%) are computed on this filtered set, the conclusion that LLM-generated vulnerability-witnessing tests 'can already provide helpful assistance' may not generalize to the broader population of real-world vulnerabilities. The paper should either provide a representativeness analysis (e.g., distributions of the filter criteria in VUL4J and in the selected subset) or explicitly restrict the conclusion to single-method-fix vulnerabilities.
  3. [Section 4.4 and Section 5.2] The context-level and ablation results are presented without any statistical testing. For example, RQ4's answer ('adding a certain amount of code context leads to significant improvements') is based on counts of 28, 34, 40, and 35 useful cases out of 50 across L0-L3, with no confidence intervals or pairwise tests. Similarly, Table 2 reports syntax rates of 66.5%, 68.5%, 62.5%, and 65.0% across four prompt configurations; these differences are small relative to the sample size and could easily be due to sampling noise. The paper should report appropriate significance tests (e.g., chi-square or Fisher's exact test for proportions) or, at minimum, confidence intervals, and should account for multiple comparisons when claiming that context or prompt changes have measurable effects.
  4. [Section 3.2 and Section 6] The evaluation uses a 'best generation' rule: if any of the up to five feedback-based generations compiles, that result is retained for the final tallies. Consequently, the reported 66.5% and 7.5% rates are per-context success rates under an oracle-like selection over repeated samples, not single-shot generation rates. The paper should report per-iteration success rates (e.g., accuracy of the first generation, and the distribution of the iteration at which the best result was obtained) so readers can interpret the effect of feedback separately from the effect of sampling multiple responses. Relatedly, the dismissal of data leakage in Section 6 cites only the authors' prior work [18]; because VUL4J was released before the GPT-4 training cutoff and contains original vulnerability-witnessing tests, the paper should either provide evidence that generated tests are not memorized versions of VUL4J tests (e.g., a similarity analysis against the dataset's tests) or state that data leakage remains an unresolved threat for the semantic-correctness rate.
minor comments (8)
  1. [Abstract] The abstract contains an unusual spacing in '7. 5%' (the period and space inside the percentage), which should be corrected.
  2. [References] Reference [18] contains a typo in the author name: 'Hegedunundefineds' should be 'Hegedűs'.
  3. [Listing 4] The phrase 'relevantexpand' appears mid-sentence in the discussion before Listing 4; this is a formatting artifact that should be fixed.
  4. [Listing 5] In Listing 5, 'Se cu ri ty Ex cep ti on' is broken by spurious spaces; the code should be presented as written Java.
  5. [Table 1] The VUL4J IDs in Table 1 are non-contiguous (e.g., VUL4J-01 through VUL4J-77 with gaps). A short note explaining that the IDs reflect the selected subset would help readers avoid assuming the numbering is sequential.
  6. [Section 4.1] The sentence 'Excluding the manually reviewed cases from the statistics' is unclear; it likely means excluding cases that were manually judged as not useful, but the wording should be revised for precision.
  7. [Section 3.3] Section 3.3 uses 'semantically correct' to describe manual labels, which conflicts with RQ2's automated semantic-correctness terminology. Using 'useful' consistently for the manual labels would reduce ambiguity.
  8. [Section 4.4] The paper cites the context-level study as 'Michele et al.' but the authors are Tufano et al.; the citation should be corrected.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline rates are externally measured outcomes, not quantities forced by the paper's own definitions or by self-citation.

full rationale

The paper's central claims are empirical measurements rather than derivations, and the measurement procedure is externally grounded. Generated tests are executed in the VUL4J Docker environment against both the vulnerable and the fixed versions; a test counts as semantically correct only if it fails before the patch and passes after it (Section 4.2), so the PASS/FAIL labels are determined by actual program behavior, not by the model's output or by the authors' definitions. Syntactic correctness is likewise judged by compilation in the real test environment, and the 68.5% 'usefulness' figure comes from two manual raters applying explicit criteria (relevance, targeting the right method, minimal human refinement, no overriding) in Section 4.3. Those criteria define what is being measured; they do not mathematically force the measured rate. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. The only self-citation is [18], used in Section 6 to assert that data leakage should not affect the results; this supports external validity but is not load-bearing for the execution-based measurements, which would stand even without that citation. Concerns about the subjectivity of 'minimal human refinement,' the lack of an inter-rater agreement statistic, and the non-random 50-case sample are correctness and generalizability risks, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper carries no mathematical derivation; its claims rest on the chosen dataset subset, the execution environment, the manual rating scale, and the assumption that GPT-4 does not simply recall VUL4J. These are domain assumptions rather than fitted parameters or invented entities.

free parameters (3)
  • max feedback iterations = 5
    The framework stops after five feedback generations without compilation errors; changing this cap changes which runs count as 'best generation' and thus the success rate.
  • consecutive compile error threshold = 3
    If three consecutive compilation errors occur, the run is abandoned; this threshold affects whether a vulnerability is counted as a generation failure.
  • model temperature = unspecified (OpenAI default)
    The paper states temperature was not specified; the default temperature is a free parameter that influences output diversity and, in turn, the syntactic/semantic correctness rates.
assumptions (5)
  • domain assumption VUL4J entries with single-class fixes are a valid proxy for real-world Java vulnerabilities.
    Used to select 50 subjects; if false, the reported percentages do not generalize beyond this convenient subset. Location: Section 3.1 Data Collection.
  • domain assumption Executing the generated test in the VUL4J Docker environment and observing FAIL before patch and PASS after patch correctly identifies a vulnerability-witnessing test.
    This operationalization defines semantic correctness in Section 4.2; if the environment misbehaves (e.g., tests not discovered), the measured 7.5% semantic rate is inaccurate.
  • domain assumption The manual usefulness judgment by two developers, one of whom is an author, is a valid and unbiased measure of 'usefulness'.
    The 68.5% usability figure rests entirely on this subjective rating; no inter-rater reliability metric is reported.
  • domain assumption GPT-4's training data does not encode the VUL4J vulnerabilities or their tests, so the model is generalizing rather than memorizing.
    The authors assert this in Section 6 citing their prior paper [18]; if VUL4J or similar code is memorized, the results overstate generation capability.
  • domain assumption Standard JUnit and Maven/Gradle toolchains behave as expected in the Docker images.
    The automatic evaluation relies on these tools; toolchain quirks could cause false PASS/FAIL/ERR labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generation." pith.science (2026). https://pith.science/paper/ACPNCOC6

@misc{pith2026250611559,
  author       = {Pith},
  title        = {Pith review of: Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACPNCOC6}},
  note         = {Machine review of arXiv:2506.11559}
}
read the original abstract

In the life-cycle of software development, testing plays a crucial role in quality assurance. Proper testing not only increases code coverage and prevents regressions but it can also ensure that any potential vulnerabilities in the software are identified and effectively fixed. However, creating such tests is a complex, resource-consuming manual process. To help developers and security experts, this paper explores the automatic unit test generation capability of one of the most widely used large language models, GPT-4, from the perspective of vulnerabilities. We examine a subset of the VUL4J dataset containing real vulnerabilities and their corresponding fixes to determine whether GPT-4 can generate syntactically and/or semantically correct unit tests based on the code before and after the fixes as evidence of vulnerability mitigation. We focus on the impact of code contexts, the effectiveness of GPT-4's self-correction ability, and the subjective usability of the generated test cases. Our results indicate that GPT-4 can generate syntactically correct test cases 66.5\% of the time without domain-specific pre-training. Although the semantic correctness of the fixes could be automatically validated in only 7. 5\% of the cases, our subjective evaluation shows that GPT-4 generally produces test templates that can be further developed into fully functional vulnerability-witnessing tests with relatively minimal manual effort. Therefore, despite the limited data, our initial findings suggest that GPT-4 can be effectively used in the generation of vulnerability-witnessing tests. It may not operate entirely autonomously, but it certainly plays a significant role in a partially automated process.

Figures

Figures reproduced from arXiv: 2506.11559 by the authors.

Figure 1
Figure 1. Overview of our methodology 3.1 One-time Preparation To enable automatic evaluation at a later stage, we first needed to collect and standardize the inputs of our framework [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages

  1. [18]

    Zoltán Ságodi, Gábor Antal, Bence Bogenfürst, Martin Isztin, Péter Hegedun- defineds, and Rudolf Ferenc. 2024. Reality Check: Assessing GPT-4 in Fixing Real-World Software Vulnerabilities. In Proceedings of the 28th International Con- ference on Evaluation and Assessment in Software Engineering (Salerno, Italy) (EASE ’24). Association for Computing Machin...

    arXiv 2024

  2. [1]

    Kamel Alrashedy and Abdullah Aljasser. 2023. Can LLMs Patch Security Issues? arXiv preprint arXiv:2312.00024 (2023)

    arXiv 2023

  3. [2]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang

    work page

  4. [3]

    Berkay Berabi, Alexey Gronskiy, Veselin Raychev, Gishor Sivanrupan, Victor Chi- botaru, and Martin Vechev. 2024. DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models. arXiv preprint arXiv:2402.13291 (2024)

    arXiv 2024

  5. [4]

    Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E Díaz Ferreyra. 2022. Vul4j: A dataset of reproducible java vulnerabilities geared towards the study of program repair techniques. In Proceedings of the 19th International Conference on Mining Software Repositories. 464–468

    work page 2022

  6. [5]

    Zirui Chen, Xing Hu, Xin Xia, Yi Gao, Tongtong Xu, David Lo, and Xiaohu Yang

  7. [6]

    Common Vulnerabilities and Exposures

    CVE 2024. Common Vulnerabilities and Exposures . Accessed: 2024-06-18

    work page 2024

  8. [7]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24)

    Exploiting Library Vulnerability via Migration Based Automating Test Generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 228, 12 pages. doi:10.1145/3597503. 3639583

    doi:10.1145/3597503

Show all 30 references
  1. [8]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1469–1481

    2023

  2. [9]

    Common Weaknesses Enumeration

    CWE 2024. Common Weaknesses Enumeration. Accessed: 2024-06-18

    2024

  3. [10]

    Aayush Garg, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. 2024. On the Coupling between Vulnerabilities and LLM-Generated Mutants: A Study on Vul4J Dataset. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 305–316. doi:10.1109/ICST60714....

    2024

  4. [11]

    Michael Fu, Chakkrit Kla Tantithamthavorn, Van Nguyen, and Trung Le. 2023. ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We? 2023 30th Asia-Pacific Software Engineering Conference (APSEC) (2023). doi:10.1109/apsec60848.2023.00085

    2023

  5. [12]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2023)

    2023

  6. [13]

    Vitor Guilherme and Auri Vincenzi. 2023. An initial investigation of ChatGPT unit test generation capability. In Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing (Campo Grande, MS, Brazil) (SAST ’23). Association for Computing Machinery, ...

    2023

  7. [14]

    Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large language models understand and can be enhanced by emotional stimuli.arXiv preprint arXiv:2307.11760 (2023)

    2023 arXiv

  8. [16]

    Alok Mathur, Shreyaan Pradhan, Prasoon Soni, Dhruvil Patel, and Rajeshkannan Regunathan. 2023. Automated Test Case Generation Using T5 and GPT-3. 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS) (2023). doi:10.1109/icaccs57279.2023.10112971

    2023

  9. [17]

    Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. 2024. Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis. arXiv preprint arXiv:2406.00992 (2024)

    2024 arXiv

  10. [19]

    Ipek Ozkaya. 2023. Application of large language models to software engineering tasks: Opportunities, risks, and implications. IEEE Software 40, 3 (2023), 4–8

    2023

  11. [20]

    Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy

  12. [21]

    Mohammed Latif Siddiqa, Joanna CS Santos, Ridwanul Hasan Tanvirb, Noshin Ulfatc, Fahmid Al Rifatd, and Vinicius Carvalho Lopes. 2023. An Empirical Study of Using Large Language Models for Unit Test Generation. arXiv preprint arXiv:2305.00418 (2023)

    2023 arXiv

  13. [22]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers. ArXiv abs/2009.05617 (2020). https://api.semanticscholar.org/CorpusID:235165921

    2020 arXiv

  14. [23]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering (2024). doi:10.1109/tse. 2024.3368208

    2024

  15. [24]

    Kunal Taneja and Tao Xie. 2008. DiffGen: Automated Regression Unit-Test Generation. 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (2008). doi:10.1109/ase.2008.60

    2008 doi

  16. [25]

    Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, and Yun Yang Zhenyu Chen. 2024. A Systematic Literature Review on Large Language Models for Automated Program Repair. arXiv preprint arXiv:2405.01466 (2024)

    2024

  17. [26]

    Quanjun Zhang, Chunrong Fang, Bowen Yu, Weisong Sun, Tongke Zhang, and Zhenyu Chen. 2024. Pre-Trained Model-Based Automated Software Vulnerability Repair: How Far are We? IEEE Transactions on Dependable and Secure Computing (2024). doi:10.1109/tdsc.2023.3308897

    2024

  18. [27]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-trained Language Models.2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (2023). doi:10.1109/ icse48619.2023.00129

    2023

  19. [28]

    Xin Zhou, Ting Zhang, and David Lo. 2024. Large Language Model for Vulnerabil- ity Detection: Emerging Results and Future Directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (Lisbon, Portugal) (I...

    2024

  20. [30]

    Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. A critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv preprint arXiv:2310.08879 (2023)

    2023 arXiv

  21. [2023]

    ArXiv abs/2310.02368 (2023)

    Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation. ArXiv abs/2310.02368 (2023). https://api.semanticscholar.org/ CorpusID:263620542

    2023 arXiv

  22. [2024]

    arXiv preprint arXiv:2402.09171 (2024)

    Automated unit test improvement using large language models at meta. arXiv preprint arXiv:2402.09171 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.