Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Automated Test Generation from Program Documentation Encoded in Code Comments

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating Javadoc contracts as test goals finds failures coverage-based tools miss.

desk verdict JUDOT is a genuine, honest incremental contribution — Javadoc contracts as first-class search objectives — but the 14.8% headline is measured on inferred formulas, and the paper's own 58% false-alarm rate leaves the RQ1 metric under-validated. read the letter →

arxiv 2504.21161 v1 pith:WBDPUQBL submitted 2025-04-29 cs.SE

Giovanni Denaro , Luca Guglielmo This is my paper · ORCID
classification cs.SE
keywords automatedtestgenerationsearch-basedtestingJavadoccontractoraclescodecommentsobjectivesJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the behaviors programmers document in Javadoc comments are a usable, and mostly untapped, source of test objectives. It proposes a generator that converts each documented postcondition into a search objective, hunts for inputs that satisfy the contract's guard, and then tries both to satisfy and to violate the contract's assertion; each resulting test receives a name and oracle taken from the contract. Compared with a coverage-driven search-based generator that uses the same contracts only as oracles, the approach is reported to test 87 additional contracts out of 775 (a 14.8% increase) and to expose 31 contract violations the baseline missed. Manual inspection traced those violations to 45 actual program faults, which is the practical payoff: documentation-driven testing can find real bugs that structural coverage alone tends to skip.

What carries the argument

The load-bearing object is the contract distance function. For a contract $c \equiv \bigwedge_j guard_j \Rightarrow \bigwedge_k assert_k$, JUDOT defines $DISTANCE_c(t) = (d^{in}_c(t)+d^{ret}_c(t))/2$, where $d^{in}_c$ accumulates normalized distances for satisfying all preconditions and guard terms, and $d^{ret}_c$ does the same for the assertion terms, with normalization $\|v\|=v/(1+v)$. The companion violation objective replaces the assertion sum by the product $\prod_k d^{ret}_{\neg assert_k}(t)$, so it reaches zero as soon as one conjunct of the assertion fails. These scalar functions, instantiated as evaluator programs, are fed to the search as fitness functions; the genetic search minimizes them and stops when the value is zero, producing tests that either satisfy or violate the focal contract. This machinery turns natural-language documentation into a smooth optimization landscape, and it is what lets the search set up situations that are behaviorally required but structurally incidental, such as a drawer that already contains the gift.

What would settle it

Take a fresh set of Java classes with human-annotated Javadoc, compute the inferred contract for every comment, and ask independent programmers whether the formula matches the prose; if a large share of formulas are judged wrong, then the 'tested contracts' metric measures agreement with the inference tool rather than with the documentation.

Watch

Extended reading notes

Core claim

JUDOT's central claim is that contract formulas inferred from Javadoc—preconditions on parameters and implication-style postconditions of the form guard ⇒ assertion—should be treated as first-class test objectives rather than as post-hoc oracles. For every postcondition contract the search builds two distance functions: one measures how close a candidate test is to satisfying the contract, the other how close it is to violating it; the genetic search minimizes both, and when a violating input is found that outcome is prioritized. The paper reports that on 118 Java classes this strategy steadily tests 586 of 775 postcondition contracts, including 87 contracts a coverage-driven baseline never hit, and reveals 107 contract violations, 31 more than the baseline. After manual triage, 45 of those violations correspond to actual failures of the classes under test, including defects developers later confirmed or fixed. The one-to-one mapping between tests and focal contracts also gives each test case a name derived from the Javadoc wording and an oracle from the contract assertion.

Load-bearing premise

The load-bearing premise is that the contract formulas automatically inferred from Javadoc mean what the developers meant; the paper itself reports that 62 of 107 detected violations were false alarms traceable to imprecise natural-language inference, so the whole comparison inherits that imprecision.

Editorial extensions

If this is right

  • The number of documented behaviors exercised by a test suite can be measured contract-by-contract, giving maintainers a coverage-like metric tied to intent rather than to code shape.
  • Tests generated this way come with meaningful names and assertions tied to one contract, so a failure directly identifies which documented behavior broke.
  • Because the violating-objective search explicitly tries to break each contract, failures can be detected even when no developer has written an expected output in advance.
  • The technique is complementary to structural coverage: two contracts in the benchmark were tested only by the coverage-driven baseline, suggesting a combined fitness function could test more contracts than either approach alone.
  • The false-alarm cost is real: 62 of 107 reported violations were traced to imprecise natural-language inference, so the approach shifts some effort from test generation to contract triage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If contract-testing were adopted, the natural next step would be to define a coverage metric over contracts—percentage of inferred contracts hit—and use it to prioritize which Javadoc comments need human review when the inferred formula is imprecise.
  • The contract-as-objective framing should transfer to other documentation conventions or to runtime assertion checking, since the search only needs guard and assertion formulas, not Javadoc specifically.
  • The 45 confirmed failures likely understate the bug-finding value, because failures in discontinued or rewritten projects could not be reported to developers, and because multi-contract baseline tests can mask one contract failure with another.
  • A testable extension would be to rank classes by contract density and compare the faults found per hour by contract-driven testing against mutation testing, which would tell teams when documentation-based testing is worth the investment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes JUDOT, a search-based test generator that treats Javadoc-derived contracts as first-class fitness functions rather than as post-hoc oracles. JUDOT uses JDOCTOR to infer precondition and postcondition contracts, defines distance-based objective functions for satisfying and for violating each postcondition contract, uses EvoSuite as the search backend with these contract-specific fitness functions, and post-processes generated tests to give them contract-derived names and oracles. The evaluation on 118 Java classes and 775 postcondition contracts compares JUDOT against EvoSuite+JDOCTOR, reporting 588 tested contracts (76.6%), 87 contracts uniquely tested by JUDOT, 31 additional contract violations, and, after manual inspection, 45 actual failures, with developer feedback on a subset. The central claim is that documentation-driven objectives cover behaviors that coverage-driven generators miss.

Significance. The core idea is timely and practically relevant: it moves documentation from being only a source of oracles to being the driver of the search, and it produces tests that are contextualized with meaningful names and assertions. The paper includes a replication package, manual inspection of all 107 reported violations, developer feedback on failure reports, and a clear motivating example. If the central claim holds, the 14.8% relative increase in tested contracts and the 45 confirmed failures are worthwhile results. However, the strength of the RQ1 claim is limited by the fact that the 'tested contract' metric is defined over JDOCTOR's inferred formulas, and the paper's own data show that these formulas are often imprecise for violation cases; this threat is acknowledged but not controlled for in the RQ1 measurement.

major comments (4)
  1. [Section IV-E and IV-F, Tables II and III] The RQ1 definition of a 'tested contract' counts a hit when a generated test executes the method with inputs satisfying the JDOCTOR guard and preconditions, and explicitly ignores the assertion. The paper's own Section IV-F reports that 62 of 107 contract violations (58%) were false alarms caused by imprecise natural-language inference. No manual audit or proxy validation is reported for the 588 tested contracts, including the 87 contracts uniquely hit by JUDOT. If JDOCTOR's imprecision is systematic rather than concentrated in violation-inducing formulas, a passing hit on an incorrect contract does not demonstrate that the documented behavior is under test. Please report a manual audit of a sample of passing and uniquely hit contracts, or recompute the RQ1 headline on a subset of human-validated contracts.
  2. [Section IV-D and IV-E] The statistical support for the headline 14.8% increase is thin. The 5-out-of-10 majority threshold for declaring a contract 'tested' is arbitrary, no confidence intervals or significance tests are reported, and 10 repetitions is a small sample for proportion estimates. Report confidence intervals for the hit rates (e.g., Wilson intervals) and a sensitivity analysis across stricter thresholds (e.g., 7/10 and 9/10) to show that the 14.8% difference is not an artifact of the chosen cutoff.
  3. [Section III-A and III-B] The treatment of @throws contracts is unclear and potentially undefined. The paper represents the NoMoreException contract as 'drawer.exceeds(limit) => retVal instanceof NoMoreException' even though unwrapAndSave returns boolean; if the method throws, there is no return value. The paper does not explain how the evaluator programs obtain a value for 'retVal' on exceptional exits, or how the fitness function handles thrown exceptions. Since @throws contracts are part of the postcondition set counted in Table I, this gap affects the core technique and should be clarified by describing the actual instrumentation semantics for exceptional termination.
  4. [Section IV-D] The time-budget comparison may favor JUDOT independently of the contract-specific fitness functions. JUDOT splits generation into multiple independent searches, with batches of up to 10 contract objectives, while EvoSuite+JDOCTOR is given a single continuous budget with all coverage fitness functions. Search restarts and objective decomposition can improve results by themselves, so the observed difference may not isolate the effect of contract-based objectives. Please add an ablation in which the backend is run with the same batching and time allocation but with coverage fitness functions, or explicitly discuss and control this confound.
minor comments (5)
  1. [Title page] The author name appears as 'Luca Gugliemo' in the header and as 'Luca Guglielmo' in the affiliation block; please correct the spelling.
  2. [Section IV-E] The sentence 'we report that a test generator tests the contract c, if it hits c in at least 5 out of 10 executions' should be phrased as an operational definition chosen by the authors, not as a natural reporting convention.
  3. [Section III-B] The notation for the two distance functions, DISTANCE_c(t) and DISTANCE_c(t), appears without a clear visual distinction between the contract-satisfying and contract-violating variants; please use distinct subscripts or superscripts throughout Section III.
  4. [Section III-B] When the normalization operator is applied to a sum of distances that are already normalized to [0,1], the result val/(1+val) is smaller than any individual term in the sum; a one-sentence intuition of why this is desirable would help readers.
  5. [Section II-B, Figure 2] The caption of Figure 2 does not state that test0 and test1 are deliberately invalid because they violate the documented precondition on limit; adding this note would make the illustration less confusing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: JUDOT's effectiveness claim is an empirical search-based result, not a prediction derived from its own inputs.

full rationale

The paper makes no first-principles prediction that reduces to its inputs by construction. JUDOT uses JDOCTOR (an external tool, not the authors' own prior work) to infer contract formulas from Javadoc, and then optimizes fitness functions whose zero points are contract-satisfying or contract-violating executions. The RQ1 metric counts a contract as 'tested' when guard and preconditions are satisfied, which overlaps with the din component of JUDOT's own objective function; however, this alignment is the standard design of search-based testing and is not circular. The comparison is meaningful because the EVOSUITE+JDOCTOR baseline is evaluated under the same contract-hit criterion while optimizing different, coverage-based objectives, and JUDOT nevertheless misses 187 of 775 contracts, while the baseline uniquely hits 2. The paper explicitly separates RQ1 (formula-level contract testing) from RQ2 (manual audit of natural-language fidelity), and Section III-E openly acknowledges JDOCTOR's imprecision. The 45 actual failures and developer feedback provide independent grounding beyond the fitness functions. The self-citations present (references 14–16, 24) are background on symbolic execution and search-based testing, not load-bearing premises for the central claim. No equation is defined in terms of another result, and no fitted parameter is renamed as a prediction. The main limitation is construct validity regarding JDOCTOR's formula fidelity, which the authors disclose, but that is a correctness risk, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on contract extraction accuracy, benchmark representativeness, search effectiveness, baseline fairness, and manual classification. The paper provides partial evidence for each, but the replication package was not independently executed and the false-alarm rate shows that the contract extraction assumption is only partially satisfied.

free parameters (3)
  • majority hit threshold = 5 of 10 executions
    A contract is counted as tested only if hit in at least 5 of 10 runs; this threshold is a methodological choice, not derived from a statistical model.
  • time budget per iteration = 35 * number of fitness functions seconds, clamped to 350
    Used to set equal total budgets for JUDOT and the baseline; the linear formula is ad hoc and directly affects comparability of the two approaches.
  • batch size for contract fitness functions = 10 fitness functions per Evosuite run
    Chosen to limit interference among fitness functions; no sensitivity analysis is reported.
assumptions (5)
  • domain assumption JDOCTOR-translated contracts faithfully represent the natural-language Javadoc specifications.
    The entire evaluation counts contracts as tested and violations as failures based on JDOCTOR formulas; Section IV-F shows 62/107 alarms are false due to imprecise inference, so this assumption is only partially met.
  • domain assumption The 118-class benchmark is representative of Java systems with Javadoc documentation.
    External validity is acknowledged as a threat in Section IV-H; the benchmark is taken from the JDOCTOR replication package and may not generalize to other systems or documentation styles.
  • domain assumption Search-based optimization with the defined distance functions can effectively explore the input space.
    The method's effectiveness depends on Evosuite's genetic algorithm finding inputs that satisfy contract guards; this is asserted in Section III-C and not independently verified.
  • domain assumption EVOSUITE+JDOCTOR is a fair baseline for coverage-driven test generation with contract oracles.
    The baseline is configured with all coverage fitness functions and equal time budget, per Section IV-A and IV-D, but fairness is not statistically verified and the time budget formula may favor one side.
  • domain assumption Manual classification of alarms as actual vs. false is accurate.
    RQ2 depends on the authors' manual inspection and a small set of developer reports, as described in Section IV-F; this classification is subjective and could shift the reported number of real bugs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Test Generation from Program Documentation Encoded in Code Comments." pith.science (2026). https://pith.science/paper/WBDPUQBL

@misc{pith2026250421161,
  author       = {Pith},
  title        = {Pith review of: Automated Test Generation from Program Documentation Encoded in Code Comments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBDPUQBL}},
  note         = {Machine review of arXiv:2504.21161}
}
read the original abstract

Documenting the functionality of software units with code comments, e.g., Javadoc comments, is a common programmer best-practice in software engineering. This paper introduces a novel test generation technique that exploits the code-comment documentation constructively. We originally address those behaviors as test objectives, which we pursue in search-based fashion. We deliver test cases with names and oracles properly contextualized on the target behaviors. Our experiments against a benchmark of 118 Java classes indicate that the proposed approach successfully tests many software behaviors that may remain untested with coverage-driven test generation approaches, and distinctively detects unknown failures.

Figures

Figures reproduced from arXiv: 2504.21161 by the authors.

Figure 1
Figure 1. A sample Java class with Javadoc documentation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Test cases computed with EVOSUITE for class GiftPack relation between the code elements and the possible program behaviors is coincidental, they may easily miss some behaviors that do not distinctively map to a target code element. For instance, the class GiftPack in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Test cases computed with JUDOT for class GiftPack with search based testing to generate test cases (Section III-C), and post-processes the test cases to add names and oracles based on the focal contracts (Section III-D). Below, we explain each step in detail. A. Inferring Contracts from Javadoc (via JDOCTOR) As first step, JUDOT relies on JDOCTOR [7] to analyze the Javadoc contracts documented in the class under tes… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Workflow of JUDOT capture to what extent a test case is distant from either satisfying or violating the contract c, respectively. Given a candidate test case tˆ, if DISTANCEc(tˆ) (resp. DISTANCEc(tˆ)) yields 0, it means that the test case tˆ satisfies (resp. violates) …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SONAR: Task-Aware Code Summary Evaluation for LLM Consumers Without References

    cs.SE 2026-08 conditional novelty 7.0 of 10

    SONAR evaluates code summaries without reference summaries by regenerating code from the summary, and shows that only correctness and abstraction strongly predict LLM performance on downstream software engineering tasks.

Reference graph

Works this paper leans on

51 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    W. J. Savitch, Absolute java. Pearson, 2016

    work page 2016

  2. [2]

    Sommerville, Software Engineering, 9/E

    I. Sommerville, Software Engineering, 9/E . Pearson Education India, 2011

    work page 2011

  3. [3]

    How to Write Doc Comments for the Javadoc Tool

    Oracle, “How to Write Doc Comments for the Javadoc Tool.” [Online]. Available: https://www.oracle.com/technical-resources/ articles/java/javadoc-tool.html

    work page

  4. [4]

    perlpod - the Plain Old Documentation format - Perldoc Browser

    MetaCPAN, “perlpod - the Plain Old Documentation format - Perldoc Browser.” [Online]. Available: https://perldoc.perl.org/perlpod

    work page

  5. [5]

    Doxygen homepage

    “Doxygen homepage.” [Online]. Available: https://www.doxygen.nl/

    work page

  6. [6]

    [Online]

    Eclipse ide. [Online]. Available: https://eclipseide.org/

    work page

  7. [7]

    Translating code comments to procedure spec- ifications,

    A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezz `e, and S. Delgado Castellanos, “Translating code comments to procedure spec- ifications,” in Proceedings of the International Symposium on Software Testing and Analysis, ser. ISSTA ’18. ACM, 2018

    work page 2018

  8. [8]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

    2014

Show all 51 references
  1. [9]

    DART: directed automated random testing,

    P. Godefroid, N. Klarlund, and K. Sen, “DART: directed automated random testing,” in Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI 2005) , 2005, pp. 213–223

    2005

  2. [10]

    JPF-SE: A Symbolic Execution Extension to Java PathFinder,

    S. Anand, C. S. P ˘as˘areanu, and W. Visser, “JPF-SE: A Symbolic Execution Extension to Java PathFinder,” in International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS), 2007, pp. 134–138

    2007

  3. [11]

    Feedback-directed random test generation,

    C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in Proceedings of the International Conference on Software Engineering , ser. ICSE ’07. ACM, 2007, pp. 75–84

    2007

  4. [12]

    Whole Test Suite Generation,

    G. Fraser and A. Arcuri, “Whole Test Suite Generation,” IEEE Trans- actions on Software Engineering , vol. 39, no. 2, pp. 276–291, 2013

    2013

  5. [13]

    Jpf-Se: A symbolic execution extension to Java pathfinder,

    S. Anand, C. S. P ˘as˘areanu, and W. Visser, “Jpf-Se: A symbolic execution extension to Java pathfinder,” in International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS 2007) , Braga, Portugal, March 2007, pp. 134–138

    2007

  6. [14]

    Symbolic Execution of Programs with Heap Inputs,

    P. Braione, G. Denaro, and M. Pezz `e, “Symbolic Execution of Programs with Heap Inputs,” in ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2015, pp. 602–613

    2015

  7. [15]

    JBSE: A Symbolic Executor for Java Programs with Complex Heap Inputs,

    ——, “JBSE: A Symbolic Executor for Java Programs with Complex Heap Inputs,” in ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2016, pp. 1018–1022

    2016

  8. [16]

    Combining symbolic execution and search-based testing for programs with complex heap inputs,

    P. Braione, G. Denaro, A. Mattavelli, and M. Pezz `e, “Combining symbolic execution and search-based testing for programs with complex heap inputs,” in Proceedings of the International Symposium on Software Testing and Analysis, ser. ISSTA ’17. ACM, 2017, pp. 90–101

    2017

  9. [17]

    Search-Based Software Test Data Generation: A Survey,

    P. McMinn, “Search-Based Software Test Data Generation: A Survey,” Software Testing, Verification and Reliability , vol. 14, no. 2, pp. 105– 156, 2004

    2004

  10. [18]

    Evolutionary Testing of Classes,

    P. Tonella, “Evolutionary Testing of Classes,” in ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis (ISSTA) , 2004, pp. 119–128

    2004

  11. [19]

    A survey of symbolic execution techniques,

    R. Baldoni, E. Coppa, D. C. D’elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques,” ACM Computing Surveys (CSUR), vol. 51, no. 3, p. 50, 2018

    2018

  12. [20]

    Search-based software engineering: Trends, techniques and applications,

    M. Harman, S. A. Mansouri, and Y . Zhang, “Search-based software engineering: Trends, techniques and applications,” ACM Computing Surveys, vol. 45, no. 1, p. 11, 2012

    2012

  13. [21]

    Pezz `e and M

    M. Pezz `e and M. Young, Software testing and analysis: process, principles, and techniques . John Wiley & Sons, 2008

    2008

  14. [22]

    The category-partition method for specifying and generating functional tests,

    T. J. Ostrand and M. J. Balcer, “The category-partition method for specifying and generating functional tests,” Communications of the ACM, vol. 31, no. 6, pp. 676–686, 1988

    1988

  15. [23]

    Evosuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: Automatic test suite generation for object-oriented software,” in Proceedings of the European Software En- gineering Conference held jointly with the ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. ESEC/FSE ...

    2011

  16. [24]

    Automatically gen- erating test cases for safety-critical software via symbolic execution,

    E. Kurian, D. Briola, P. Braione, and G. Denaro, “Automatically gen- erating test cases for safety-critical software via symbolic execution,” Journal of Systems and Software , vol. 199, p. 111629, 2023

    2023

  17. [25]

    Whole test suite generation,

    G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Transac- tions on Software Engineering , vol. 39, no. 2, pp. 276–291, 2013

    2013

  18. [26]

    The stanford corenlp natural language processing toolkit,

    C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel, S. Bethard, and D. McClosky, “The stanford corenlp natural language processing toolkit,” in Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations , 2014, pp. 55–60

    2014

  19. [27]

    Generating typed dependency parses from phrase structure parses

    M.-C. De Marneffe, B. MacCartney, C. D. Manning et al., “Generating typed dependency parses from phrase structure parses.” in Lrec, vol. 6, 2006, pp. 449–454

    2006

  20. [28]

    Com- bining multiple coverage criteria in search-based unit test generation,

    J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri, “Com- bining multiple coverage criteria in search-based unit test generation,” in Proceedings of the 7th International Symposium on Search-Based Software Engineering, ser. SSBSE ’15. Springer, 2015, pp. 93–108

    2015

  21. [29]

    Evosuite at the sbst 2021 tool competition,

    S. V ogl, S. Schweikl, G. Fraser, A. Arcuri, J. Campos, and A. Panichella, “Evosuite at the sbst 2021 tool competition,” in 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST) . IEEE, 2021, pp. 28–29

    2021

  22. [30]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” arXiv preprint arXiv:2009.05617, 2020

    2009 arXiv

  23. [31]

    A3test: Assertion-augmented automated test case generation,

    S. Alagarsamy, C. Tantithamthavorn, and A. Aleti, “A3test: Assertion-augmented automated test case generation,” arXiv preprint arXiv:2302.10352, 2023

    2023 arXiv

  24. [32]

    What is a good test case?

    J. Cem Kaner, “What is a good test case?” Florida Institute of Technol- ogy Department of Computer Sciences , 2003

    2003

  25. [33]

    Release ISSTA 2016 · albertogoffi/toradocu

    A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezz `e, and S. Delgado Castellanos, “Release ISSTA 2016 · albertogoffi/toradocu.” [Online]. Available: https://github.com/albertogoffi/toradocu/releases/ tag/v0.1

    2016

  26. [34]

    toradocu/src/test/resources/goal-output at master · albertogoffi/- toradocu · GitHub

    ——, “toradocu/src/test/resources/goal-output at master · albertogoffi/- toradocu · GitHub.” [Online]. Available: https://github.com/albertogoffi/ toradocu/tree/master/src/test/resources/goal-output

  27. [35]

    Jbmc: A bounded model checking tool for verifying java bytecode,

    L. Cordeiro, P. Kesseli, D. Kroening, P. Schrammel, and M. Trtik, “Jbmc: A bounded model checking tool for verifying java bytecode,” in International Conference on Computer Aided Verification . Springer, 2018, pp. 183–190

    2018

  28. [36]

    Pex: White box test generation for .NET,

    N. Tillmann and J. de Halleux, “Pex: White box test generation for .NET,” in Proceedings of the 2nd International Conference on Tests and Proofs , ser. TAP’08. Berlin, Heidelberg: Springer-Verlag, 2008, pp. 134–153. [Online]. Available: http://portal.acm.org/citation.cfm?id= 1...

    2008

  29. [37]

    Assertion-oriented automated test data generation,

    B. Korel and A. M. Al-Yami, “Assertion-oriented automated test data generation,” in Proceedings of IEEE 18th International Conference on Software Engineering. IEEE, 1996, pp. 71–80

    1996

  30. [38]

    Cute and jcute: Concolic unit testing and explicit path model-checking tools: (tool paper),

    K. Sen and G. Agha, “Cute and jcute: Concolic unit testing and explicit path model-checking tools: (tool paper),” in Computer Aided Verification: 18th International Conference, CAV 2006, Seattle, WA, USA, August 17-20, 2006. Proceedings 18 . Springer, 2006, pp. 419– 423

    2006

  31. [39]

    Automated test generation for activation of assertions in rtl models,

    Y . Lyu and P. Mishra, “Automated test generation for activation of assertions in rtl models,” in 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC). IEEE, 2020, pp. 223–228

    2020

  32. [40]

    Checking java assertions using automated test-case generation,

    R. Caballero, M. Montenegro, H. Kuchen, and V . von Hof, “Checking java assertions using automated test-case generation,” in Logic-Based Program Synthesis and Transformation: 25th International Symposium, LOPSTR 2015, Siena, Italy, July 13-15, 2015. Revised Selected Papers

    2015

  33. [41]

    Springer, 2015, pp. 221–226

    2015

  34. [42]

    Automated program flaw finding using simulated annealing,

    N. Tracey, J. Clark, and K. Mander, “Automated program flaw finding using simulated annealing,” in Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis , ser. ISSTA ’98. New York, NY , USA: ACM, 1998, pp. 73–81. [Online]. Available: http...

    1998

  35. [43]

    Pseudo-oracles for non-testable programs,

    M. D. Davis and E. J. Weyuker, “Pseudo-oracles for non-testable programs,” in Proceedings of the ACM’81 Conference , 1981, pp. 254– 257

    1981

  36. [44]

    Generating diverse software versions with genetic program- ming: an experimental study,

    R. Feldt, “Generating diverse software versions with genetic program- ming: an experimental study,”IEE Proceedings-Software, vol. 145, no. 6, pp. 228–236, 1998

    1998

  37. [45]

    Search-based failure discovery using testability transfor- mations to generate pseudo-oracles,

    P. McMinn, “Search-based failure discovery using testability transfor- mations to generate pseudo-oracles,” in Proceedings of the 11th Annual conference on Genetic and evolutionary computation , 2009, pp. 1689– 1696

    2009

  38. [46]

    Automatic system testing of programs without test oracles,

    C. Murphy, K. Shen, and G. Kaiser, “Automatic system testing of programs without test oracles,” in Proceedings of the eighteenth interna- tional symposium on Software testing and analysis , 2009, pp. 189–200

    2009

  39. [47]

    Using jml runtime assertion checking to automate metamorphic testing in applications without test oracles,

    ——, “Using jml runtime assertion checking to automate metamorphic testing in applications without test oracles,” in 2009 International Conference on Software Testing Verification and Validation . IEEE, 2009, pp. 436–445

    2009

  40. [48]

    Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing,

    R. Guderlei and J. Mayer, “Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing,” in Seventh International Conference on Quality Software (QSIC 2007) . IEEE, 2007, pp. 404–409

    2007

  41. [49]

    Inferring better contracts,

    Y . Wei, C. A. Furia, N. Kazmin, and B. Meyer, “Inferring better contracts,” in Proceedings of the International Conference on Software Engineering, ser. ICSE ’11. ACM, 2011, pp. 191–200

    2011

  42. [50]

    Stateful testing: Finding more errors in code and contracts,

    Y . Wei, H. Roth, C. A. Furia, Y . Pei, A. Horton, M. Steindorfer, M. Nordio, and B. Meyer, “Stateful testing: Finding more errors in code and contracts,” in Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’11. Washin...

    2011

  43. [443]

    Available: http://dx.doi.org/10.1109/ASE.2011.6100094

    [Online]. Available: http://dx.doi.org/10.1109/ASE.2011.6100094

    2011

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.