Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Assertify claims that LLMs, guided by context-rich prompts with few-shot examples, can generate production assertions for Java code that are syntactically accurate up to 97.4%, statically meaningful up to 83.5%, and structurally similar…

desk verdict A useful new dataset and task framing for LLM-based production assertion generation, but the evaluation stops at compilability and lexical similarity—no runtime validity—and same-repo few-shot examples may inflate the similarity numbers. read the letter →

arxiv 2411.16927 v1 pith:3OO7OJ34 submitted 2024-11-25 cs.SE cs.AI

Mohammad Jalili Torkamani , Abhinav Sharma , Nikita Mehrotra , Rahul Purandare This is my paper · ORCID
classification cs.SEcs.AI
keywords productionassertionslargelanguagemodelspromptengineeringfew-shotlearningJavastaticsemanticsROUGE-Lassertiongeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models, given carefully assembled context, can generate production assertions directly inside Java source code, a task distinct from generating unit-test assertions. It introduces Assertify, a pipeline that extracts method context, builds prompts with code summaries, input/output details, and few-shot examples, then inserts model-generated assertions at predicted line numbers. On 983 methods from 22 mature Java repositories, the best configuration achieves syntactic accuracy up to 97.4%, static semantic accuracy up to 83.5%, and ROUGE-L structural similarity of 0.526 against developer-written assertions. If these numbers hold beyond the corpus, automation could relieve developers of a manual, error-prone task and improve code reliability.

What carries the argument

The carrying mechanism is a context-enrichment pipeline rather than a new model or loss function. Assertify strips assertions and comments from a candidate method, then builds a prompt that includes the method name and signature, a generated code summary, input/output descriptions, summaries of internally invoked methods, and the top three similar methods selected by cosine similarity over method vectors for few-shot learning. The LLM returns assertion-plus-line-number pairs, which a postprocessor filters by checking that the line numbers fall within the method's braces; the repository is compiled after insertion to detect static semantic errors, and ROUGE-L measures structural similarity.

What would settle it

Run Assertify on methods from Java repositories that are not in the original 22-repository corpus, using the same prompt template and few-shot examples drawn only from the original corpus, and compare the average ROUGE-L to the reported 0.526; a drop toward the no-few-shot baseline would show the similarity is corpus-specific.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that prompt engineering with few-shot learning is what makes LLM-generated production assertions resemble developer-written ones. It shows that each added context layer—code summary, input/output description, invoked-method summaries, and similar methods—raises structural similarity and reduces syntactic and static-semantic errors, with the fullest prompt giving the best results. The paper also reports that generated assertions are typically equal to or stronger than the original assertions, and that few-shot learning specifically increases the frequency of exact assertion-set matches.

Load-bearing premise

The few-shot examples and the evaluation methods are drawn from the same 22 repositories, so the high structural similarity may reflect the model imitating in-corpus assertion style rather than generalizing to unseen projects.

Editorial extensions

If this is right

  • Production assertions can be generated automatically in Java codebases without running tests or needing a test prefix, which existing unit-test assertion generators require.
  • The best configuration uses the richest prompt plus few-shot examples; removing few-shot examples costs the most accuracy in the ablation study.
  • Generated assertions tend to be equal to or stronger than developer-written assertions, not merely weaker approximations.
  • Model choice trades cost and latency against accuracy, with the strongest model giving the top scores while cheaper models remain viable for developers with tighter budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported ROUGE-L similarity likely overstates generalization because the few-shot examples and the evaluation methods come from the same 22 repositories; a held-out repository test would be a stricter measure.
  • The static semantic check verifies only that the code compiles, not that the assertion is true at runtime, so the 83.5% figure is an upper bound on the share of assertions that are dynamically meaningful.
  • The same pipeline should transfer to other statically typed languages, but few-shot examples would need to be drawn from the target language; that transfer is a testable extension the paper leaves open.
  • If few-shot examples were taken from a separate corpus, the measured gain from few-shot learning would cleanly separate genuine generalization from in-corpus style mimicry.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Assertify, a pipeline that uses large language models (GPT-3, GPT-4, GPT-4o) with context-rich prompts and few-shot learning to generate production-code assertions in Java. The approach extracts method metadata, generates code summaries, includes input/output descriptions and summaries of invoked methods, and retrieves the top three similar methods as few-shot examples. The evaluation is performed on 983 methods from 22 GitHub repositories and measures syntactic parse success, static semantic success via repository compilation, and ROUGE-L similarity to developer-written assertions. The authors report syntactic accuracy up to 97.4%, static semantic accuracy up to 83.5%, and an average ROUGE-L score of 0.526, concluding that few-shot learning improves structural similarity to original assertions.

Significance. If the reported results are robust, Assertify would address a genuine gap: most prior work generates assertions for unit tests, whereas production assertions serve documentation and runtime-checking purposes. The paper contributes a reusable dataset of 2,810 methods with developer-written assertions, a modular end-to-end tool, and prompt-ablation experiments across three GPT models, with artifacts released for reproducibility. The syntactic and static-semantic accuracy results, even with the study's limitations, provide a useful data point for applying LLMs to production-code instrumentation. However, the central RQ3 claim of structural resemblance to developer assertions rests on a single metric and an evaluation design that may leak project-specific style into the few-shot prompts; the manuscript needs stronger validation before the headline similarity claim is accepted.

major comments (4)
  1. [§6.1, §5 (Similar Method Extractor)] The few-shot learning dataset and the evaluation set are both drawn from the same 22 repositories. Because the Similar Method Extractor selects the top three cosine-similar methods from the FSL set without excluding examples from the candidate's own repository, the model can be prompted with methods that share project-specific naming conventions, idioms, and assertion style. Since RQ3 is answered exclusively with ROUGE-L, the reported 0.526 score may reflect in-corpus mimicry rather than generalizable resemblance to developer-written assertions. The paper should evaluate on a repository-held-out split, or at least report the fraction of selected few-shot examples coming from the same repository as the candidate and analyze ROUGE-L conditioned on that factor.
  2. [§7 (RQ3)] The results section is internally inconsistent about which model achieves the best ROUGE-L score. The text first states that scores range "from 0.374 for GPT-3 up to 0.526 for GPT-4," then says Assertify "performs best (overall) when using the GPT-4o model," and finally gives "the best ROUGE-L similarity of 0.52 and 0.51 with the GPT-4 and GPT-4o models, respectively." The abstract reports 0.526 without model attribution. These statements cannot all be correct, and the discrepancy directly affects the paper's central claim. Please report the exact per-model, per-prompt ROUGE-L values and reconcile the abstract with Section 7.
  3. [§6.2 (SME), §7 (RQ2)] The static semantic evaluation only checks whether the repository compiles after inserting the generated assertions; it does not execute the code or validate that the asserted boolean conditions actually hold at runtime. The authors acknowledge this limitation, but the phrasing of RQ2 and several results paragraphs (e.g., "assertions free of syntactic and static semantic errors") may lead readers to infer semantic correctness. Because an assertion that compiles but is always false is worse than useless in production, the claims should be strictly limited to "syntactically valid and compilable" unless a dynamic evaluation is added, even on a subset of the dataset.
  4. [§7 (all RQs)] All reported figures are point estimates from a single inference run per configuration, with no error bars, confidence intervals, or repeated runs. Given that LLM APIs are stochastic, statements such as "few-shot learning increases the ROUGE-L score by 0.09 with GPT-4" are not statistically supported. The paper should either run each configuration multiple times and report variance, or explicitly frame the results as a single-run exploratory observation rather than a comparative evaluation.
minor comments (5)
  1. [§7 (RQ3)] The phrase "as it receives more information about the model" appears to be a typo; the intended meaning is likely "more information about the method" or "about the code."
  2. [References] References [7] and [8] are the same paper (TOGA), and references [36] and [37] are also the same paper; please deduplicate.
  3. [§2, Reference [6]] The text uses "SourceGraph" while the reference uses "Sourcegraph"; unify the spelling.
  4. [§6.2] The SNE rate is defined, but the results report "syntactic accuracy" and "static semantic accuracy" without explicitly stating that accuracy equals one minus the corresponding error rate; stating this relationship at the metric definitions would improve readability.
  5. [§7 (RQ3), Figure 7] The caption and axis labels of Figure 7 are not described in the text; please ensure the figure clearly distinguishes per-model trend lines and indicates whether the reported values are ROUGE-L F1 scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is empirical and the ROUGE-L claim is not definitionally tied to the prompt inputs.

full rationale

Assertify's central claims are empirical measurements: syntactic accuracy, static semantic accuracy, and ROUGE-L similarity between generated and developer-written assertions. The ROUGE-L score is computed against original assertions in the evaluation set, and the few-shot examples come from a separate FSL set; the metric is not defined in terms of the prompt contents, and no parameter is fitted to the reported outcome. The 0.5 similarity threshold and dataset filters are stated as design choices, not fitted values. The paper does not invoke a load-bearing self-citation or a uniqueness theorem, and the approach is not an ansatz smuggled in via citation. The same-repository origin of the FSL and evaluation sets is a legitimate threat to external validity and could inflate similarity through style leakage, but it is an experimental confound rather than a circular derivation: the model could still produce assertions that differ from the reference, and the reported score is not forced by construction. Thus no step in the paper's reasoning reduces to its own inputs by definition or by a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on LLM capability, the validity of original assertions as ground truth, build-based semantic checking, and ROUGE-L as a quality metric. No physical or formal entities are introduced.

free parameters (4)
  • similar_method_similarity_threshold = 0.5
    Cosine similarity threshold for selecting few-shot examples (Section 5). Chosen by the authors to 'strike a balance between precision and recall' without tuning; affects which examples the LLM sees and thus ROUGE-L.
  • few_shot_example_count = 3
    Top three similar methods are included in the prompt (Section 5). This choice is justified by token limits, not by experiments.
  • repository_filter_stars = 500
    Minimum stars for repository inclusion (Section 6.1). Thresholds set after analyzing distribution, i.e., post hoc, affecting dataset composition.
  • repository_filter_assertions = 50
    Minimum number of production assertions per repository (Section 6.1). Also set post hoc based on distribution.
assumptions (4)
  • domain assumption LLMs (GPT-3.5, GPT-4, GPT-4o) are capable of generating syntactically and semantically valid production assertions when given sufficient context.
    The entire approach relies on this; the paper tests it empirically but does not prove it.
  • domain assumption Original developer-written assertions in the corpus are correct, representative, and a valid target for generation.
    Used as ground truth for ROUGE-L and equality comparisons (Section 6.2.3). If original assertions are idiosyncratic or wrong, similarity to them is not a quality measure.
  • domain assumption Compiling the repository with the replaced method detects static semantic errors introduced solely by the assertions.
    The SME metric compiles the whole repository (Section 6.2); this assumes no pre-existing build failures and that the build is reproducible in the evaluation environment.
  • domain assumption ROUGE-L longest-common-subsequence is an appropriate measure of structural similarity for assertions.
    The paper chooses ROUGE-L over ROUGE-N because assertions are expression-level; this is a modeling choice, not an established benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code." pith.science (2026). https://pith.science/paper/3OO7OJ34

@misc{pith2026241116927,
  author       = {Pith},
  title        = {Pith review of: ASSERTIFY: Utilizing Large Language Models to Generate Assertions for Production Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OO7OJ34}},
  note         = {Machine review of arXiv:2411.16927}
}
read the original abstract

Production assertions are statements embedded in the code to help developers validate their assumptions about the code. They assist developers in debugging, provide valuable documentation, and enhance code comprehension. Current research in this area primarily focuses on assertion generation for unit tests using techniques, such as static analysis and deep learning. While these techniques have shown promise, they fall short when it comes to generating production assertions, which serve a different purpose. This preprint addresses the gap by introducing Assertify, an automated end-to-end tool that leverages Large Language Models (LLMs) and prompt engineering with few-shot learning to generate production assertions. By creating context-rich prompts, the tool emulates the approach developers take when creating production assertions for their code. To evaluate our approach, we compiled a dataset of 2,810 methods by scraping 22 mature Java repositories from GitHub. Our experiments demonstrate the effectiveness of few-shot learning by producing assertions with an average ROUGE-L score of 0.526, indicating reasonably high structural similarity with the assertions written by developers. This research demonstrates the potential of LLMs in automating the generation of production assertions that resemble the original assertions.

Figures

Figures reproduced from arXiv: 2411.16927 by the authors.

Figure 1
Figure 1. Assertify Workflow. path in the expected response during few-shot prompting, our approach enriches the input prompt and resembles how developers craft production assertions. In this section, we present a detailed perspective on our methodology, adopting the previous code fragment (Listing 5) to explain the entire process [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Example of Prompt Creation. explanations. This code summary ensures that the information integrated into the prompt helps the LLM understand the method’s intended behavior (Step 2). Going a step further, Assertify also incorporates information about the method’s inputs and outputs into the prompt. This additional data enables the LLM to consider the expected behavior of the method, just as developers do manually whe… view at source ↗
Figure 3
Figure 3. SNE Rate [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Syntactic Accuracy [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 7
Figure 7. Figure 7: Average ROUGE-L Score [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: Response Time [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

Reference graph

Works this paper leans on

43 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Paheli Bhattacharya, Manojit Chakraborty, Kartheek N S N Palepu, Vikas Pandey, Ishan Dindorkar, Rakesh Rajpurohit, and Rishabh Gupta. 2023. Exploring Large Language Models for Code Explanation. arXiv:2310.16673 [cs.SE]

    arXiv 2023

  2. [2]

    Doga Cambaz and Xiaoling Zhang. 2024. Use of AI-driven Code Generation Models in Teaching and Learning Programming: a Systematic Literature Review. InProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1(<conf-loc>, <city>Portland</city>, <state>OR</state>, <country>USA</country>, </conf-loc>)(SIGCSE 2024). Association for Com...

    arXiv 2024

  3. [3]

    Hugo Caselles-Dupré, Florian Lesaint, and Jimena Royo-Letelier. 2018. Word2Vec applied to Recommendation: Hyperparameters Matter. arXiv:1804.04212 [cs.IR]

    work page Pith review arXiv 2018

  4. [4]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Trans. Intell. Syst. Technol. 15, 3, Article 39 (mar 2024), 45 pages. https://doi.org/10.1145/3641289

    doi:10.1145/3641289 2024

  5. [5]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2023. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review

    work page 2023

  6. [6]

    Sourcegraph Company. 2013. Repositories dataset extraction source . Retrieved Aug 20, 2023 from https://sourcegraph. com/ Assertify: Utilizing Large Language Models to Generate Assertions for Production Code 19

    work page 2013

  7. [9]

    Minh Ngoc Dinh, David Abramson, Donny Kurniawan, Chao Jin, Bob Moench, and Luiz DeRose. 2011. Assertion Based Parallel Debugging. In 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing . 63–72. https://doi.org/10.1109/CCGrid.2011.44

    work page doi:10.1109/ccgrid.2011.44 2011

  8. [10]

    Bassem Elkarablieh, Ivan Garcia, Yuk Lai Suen, and Sarfraz Khurshid. 2007. Assertion-based repair of complex data structures. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1321631.1321643

    arXiv 2007

Show all 43 references
  1. [11]

    Michael D. Ernst. 2000.Dynamically Discovering Likely Program Invariants. Ph.D. University of Washington Department of Computer Science and Engineering, Seattle, Washington

    2000

  2. [12]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE ’11). Association...

    2011

  3. [13]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yin, Daxin Jiang, and Ming Zhou. 2020. GraphCodeBERT: Pre-training Code Representations with Data Flow. https://doi.org/10.48550 /arXiv.2009.08366

    2020 doi arXiv

  4. [14]

    Hambarde and Hugo Proença

    Kailash A. Hambarde and Hugo Proença. 2023. Information Retrieval: Recent Advances and Beyond. IEEE Access 11 (2023), 76581–76604. https://doi.org/10.1109/access.2023.3295776

    2023

  5. [15]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE ’23). IEEE Press, 2312–23...

    2023

  6. [16]

    Bogdan Korel and Ali M. Al-Yami. 1996. Assertion-oriented automated test data generation. In Proceedings of the 18th International Conference on Software Engineering (Berlin, Germany) (ICSE ’96). IEEE Computer Society, USA, 71–80

    1996

  7. [17]

    Gunnar Kudrjavets, Nachiappan Nagappan, and Thomas Ball. 2006. Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation. Proceedings - International Symposium on Software Reliability Engineering, ISSRE, 204–212. https://doi.org/10.1109/ISSR...

    2006 doi

  8. [18]

    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas

  9. [19]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  10. [20]

    Vadim Liventsev, Anastasiia Grishina, Aki Härmä, and Leon Moonen. 2023. Fully Autonomous Programming with Large Language Models. In Proceedings of the Genetic and Evolutionary Computation Conference . ACM. https: //doi.org/10.1145/3583131.3590481

    2023

  11. [21]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen

  12. [22]

    Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating Diverse Code Explanations using the GPT-3 Large Language Model. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2(Lugano and ...

    2022

  13. [23]

    In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V

    Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Develop- ment E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (<conf-loc>, <city>Toronto ON</city>, <country>Canada</country>, </conf-loc...

    2023

  14. [24]

    Thanhvu Nguyen, Deepak Kapur, Westley Weimer, and Stephanie Forrest. 2014. DIG: A Dynamic Invariant Generator for Polynomial and Array Invariants. ACM Transactions on Software Engineering and Methodology 23 (09 2014). https://doi.org/10.1145/2556782

    2014 doi

  15. [25]

    Mocha maintainers. 2023. Mocha - the fun, simple, flexible JavaScript test framework. https://mochajs.org/

    2023

  16. [26]

    Archit Parnami and Minwoo Lee. 2022. Learning from Few Examples: A Summary of Approaches to Few-Shot Learning. https://doi.org/10.48550/arXiv.2203.04291 20 Mohammad Jalili Torkamani, Abhinav Sharma, Nikita Mehrotra, and Rahul Purandare

    2022 doi arXiv

  17. [27]

    Carlos Pacheco and Michael D. Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion (Montreal, Quebec, Canada) (OOPSLA ’07) . Association for Computing ...

    2007

  18. [28]

    Rosenblum

    D.S. Rosenblum. 1995. A practical approach to programming with assertions.IEEE Transactions on Software Engineering 21, 1 (1995), 19–31. https://doi.org/10.1109/32.341844

    1995 doi

  19. [29]

    Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21) . Association for Computing Machinery, New ...

    2021

  20. [30]

    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (ICER 2022) . ACM. ...

    2022

  21. [31]

    Rosenblum

    David S. Rosenblum. 1992. Towards a method of programming with assertions. In Proceedings of the 14th International Conference on Software Engineering (Melbourne, Australia) (ICSE ’92). Association for Computing Machinery, New York, NY, USA, 92–104. https://doi.org/10.1145/143...

    1992

  22. [32]

    Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020. Evolutionary improvement of assertion oracles (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1178–1189. https://doi.org/10.1 145/3368089.3409758

    2020

  23. [33]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. arXiv:2302.06527 [cs.SE]

    2023 arXiv

  24. [34]

    Hailong Wang, Tongtong Xu, and Bei Wang. 2024. Deep Multiple Assertions Generation. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (Lisbon, Portugal) (FORGE ’24). Association for Computing Machinery, New York...

    2024

  25. [35]

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Deng, and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers. https://doi.org/10.48550/arXiv.2009.05617

    2020 doi arXiv

  26. [36]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE ’20) . ACM. https://doi.org/10.1145/3...

    2020

  27. [37]

    Kwok, and Lionel M

    Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a Few Examples: A Survey on Few-shot Learning. ACM Comput. Surv. 53, 3, Article 63 (June 2020), 34 pages. https://doi.org/10.1145/3386252

    2020 doi

  28. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou

  29. [39]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On Learning Meaningful Assert Statements for Unit Test Cases. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) . 1398–1409

    2020

  30. [40]

    Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023. SatLM: Satisfiability-Aided Language Models Using Declarative Prompting. arXiv:2305.09656 [cs.CL]

    2023 arXiv

  31. [41]

    arXiv:2201.11903 [cs.CL]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL]

    arXiv

  32. [42]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer- Smith, and Douglas Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. https://doi.org/10.48550/arXiv.2302.11382

    2023 doi arXiv

  33. [44]

    Hao Yu, Yiling Lou, Ke Sun, Dezhi Ran, Tao Xie, Dan Hao, Ying Li, Ge Li, and Qianxiang Wang. 2022. Automated Assertion Generation via Information Retrieval and Its Integration with Deep Learning. In Proceedings of the 44th International Conference on Software Engineering (Pitt...

    2022

  34. [45]

    Lucas Zamprogno, Braxton Hall, Reid Holmes, and Joanne M. Atlee. 2023. Dynamic Human-in-the-Loop Assertion Generation. IEEE Transactions on Software Engineering 49, 4 (2023), 2337–2351. https://doi.org/10.1109/TSE.2022.321 7544

    2023 doi

  35. [2023]

    arXiv:2304.03938 [cs.CY]

    Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938 [cs.CY]

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.