Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that the next big open problem in automated test generation is producing tests that catch bugs in a pull request just before it lands, and that LLMs' ability to read code and natural language makes the problem tractable.

desk verdict The Catching JiTTest Challenge is a real and impactful research target, but the formal definitions and the oracle-agreement assumption need work before they can carry the weight this paper puts on them. read the letter →

arxiv 2504.16472 v2 pith:N6YZFQUZ submitted 2025-04-23 cs.SE cs.AI

Mark Harman , Peter O'Hearn , Shubho Sengupta This is my paper · ORCID
classification cs.SEcs.AI
keywords just-in-timetestinghardeningtestscatchingregressionoraclescavengingLLM-basedtestgenerationpullrequestprecisionandrecallin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to sharpen two fundamental notions in automated software testing: hardening tests, which pass on the current code and guard against future regressions, and catching tests, which fail on the current change and expose a bug in it. It argues that mainstream test generation has fallen into a 'Regression Only Trap' by relying on the regression oracle, so it cannot catch bugs in new functionality. The central open problem it formulates is the Catching JiTTest Challenge: given a buggy pull request and its parent revision, automatically generate a test that reveals the pull request's bug with low risk of failing on non-buggy pull requests. A sympathetic reader would care because the paper shows this challenge is well-posed, currently untackled, and that solving it would also yield a way to uncover latent bugs in legacy code by deleting and re-inserting code as a fake pull request. The paper grounds its optimism in the ability of large language models to scavenge intended behaviour from non-executable text such as pull request titles and comments.

What carries the argument

The load-bearing machinery is a family of definitions built on a single object: a partial oracle Oracle_R(t) that states whether test t is expected to pass or fail on revision R, together with the predicates Builds, Passes, and Fails. Hardening tests are defined through the regression oracle: they pass on the current revision and fail correctly on some child revision. Catching tests are defined by failure on the current revision, with strong catching requiring the oracle to call that failure correct. The eight-cell classification (weak/strong hardening on the parent, weak/strong catching on the child) is what organises the paper's deployment analysis: it maps each cell to a decision—land, report signal, or discard. The Just-in-Time definition is the other key object: a JiTTest for revision R is timely for R but not timely for its parent, which means it can be generated after the pull request appears but before first human review.

What would settle it

Take a set of real pull requests with known bugs and a set of clean ones from the same repository; run a JiTTest generator that sees only the pull request, its parent, and the pull request's own text; then measure whether its failing tests distinguish the two sets at the promised precision. If no generator can beat a random-or-trivial baseline without false positives, the premise that the challenge is ready to be solved collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a classification plus a challenge. Using an oracle that says whether a given test should pass or fail on a given revision, the paper defines weak and strong hardening tests (passing now and correctly failing on some future revision) and weak and strong catching tests (failing now, correctly or not). Crossing hardening on the parent with catching on the child yields eight behavioural categories, of which only three give reliable signals; five are misleading. The paper's headline claim is that the Catching JiTTest Challenge—generate, just before a pull request lands, a test that fails on that pull request when it is buggy and does not fail on non-buggy ones—is the most impactful currently open problem in automated test generation. It further claims that any solution can be repurposed: delete a code fragment, re-insert it as a pull request, generate a catching JiTTest, and the same machinery finds latent bugs already in production. The paper proposes measuring success by recall at fixed precision, R@P=p, rather than by aggregates that trade precision against recall.

Load-bearing premise

The whole framework depends on engineers agreeing, for each revision and test, on whether the test's expected outcome is pass or fail; if one engineer calls a failure a bug and another calls it intended behaviour, the eight-way classification and the Catching JiTTest Challenge lose their precision.

Editorial extensions

If this is right

  • A solution to the Catching JiTTest Challenge would let automated testing catch bugs in new functionality, not just regressions, breaking the Regression Only Trap.
  • Any such solution doubles as a latent-bug finder: delete-and-reinsert turns arbitrary legacy code into a fresh pull request for the generator to test.
  • Running a generated JiTTest on the parent revision is cheap and gives extra signal; the paper concludes this should become standard practice.
  • Failing JiTTests that also fail on the parent should usually be discarded unless new techniques can separate true bugs from false positives.
  • Human review of weak hardening tests turns them into strong hardening tests, removing three of the five misleading signal categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the JiTTest framework is adopted, precision-at-threshold (R@P=p) could replace F1-style aggregation as the default metric for industrial test generators, because it encodes the asymmetry that false positives are costlier than missed bugs.
  • Oracle scavenging from non-executable text suggests a testable benchmark: measure whether giving an LLM only a parsimonious pull request title and code comments yields higher catch rates than giving it the code alone.
  • The delete-and-reinsert construction implies that a mature codebase's history of reverts or bug-fix commits could be mined to create a labelled evaluation set for catching JiTTests, without needing newly injected faults.
  • The paper's oracle caveat points to a social extension: in contested behaviours, the JiTTest system could report the disagreement rather than a pass/fail verdict, turning the oracle friction into a requirements-elicitation signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper formalizes two kinds of automated tests in a revision-tree setting: hardening tests, which pass on the current revision and are intended to catch future regressions, and catching tests, which fail on the current revision and may reveal bugs in it. It defines weak and strong variants of each, distinguishes regression-catching from functionality-catching tests, and presents an eight-way classification of test signals (Section 4). The main proposal is the Catching JiTTest Challenge (Section 6): given a buggy pull request and its parent, automatically generate a test that can reveal bugs in the pull request with low risk of false-positive failure on non-buggy pull requests. The paper also analyzes deployment options for timely and just-in-time tests (Section 5), and argues that a solution to the challenge can be repurposed to find latent bugs in legacy code by deleting and re-inserting code as a synthetic pull request (Section 6.5). The paper is written as an FSE 2025 keynote companion and draws on the authors' industrial experience with LLM-based test generation at Meta.

Significance. The paper's central conceptual contribution is to identify and name the Catching JiTTest Challenge, which is a plausible and practically important gap in automated test generation: most current techniques harden against future regressions, while catching bugs in the pull request being reviewed is comparatively under-addressed. The deployment tables (Tables 1-3) provide a useful decision-oriented taxonomy, and the delete-and-reinsert argument (Section 6.5) is an elegant reduction that broadens the challenge's impact. The paper is also appropriately candid about the oracle problem and the social nature of software engineering, and the 'oracle scavenging' idea is a promising research direction. However, the claimed 'Precise Formal Foundations' currently contain several errors: Definition 9 is not a well-formed formula, and Definitions 12 and 14 contain an apparently trivial existential condition. These must be corrected before the definitions can be used as stated. The well-posedness of the Catching JiTTest Challenge is conditional on the oracle-agreement assumption, which should be stated explicitly.

major comments (3)
  1. [3.4] Definition 9, the definition of Perfect Recall Hardening Test, is not a well-formed formula: the first occurrence of R' in 'Oracle_R'(t)=fail' is free, while the second occurrence is bound by the universal quantifier in '∀R'.Fails_R'(t)'. The intended perfect-recall property is presumably Hardening_R(t) ∧ ∀R' . (Oracle_R'(t)=fail ⇒ Fails_R'(t)). As written, the definition cannot be used to derive the 'perfect precision and recall' conjunction, and this undermines the paper's claim of precise formal foundations.
  2. [3.7 and 3.8] Definitions 12 and 14 both contain the existential clause '∃R'.Parent(R)=Parent(R)', which is trivially true because Parent(R) is identical to itself. The intended condition is '∃R'.Parent(R')=Parent(R)'. In Definition 12 this error reduces a weak regression catching test to merely 'passes on parent and fails on R', dropping the requirement that the test be a hardening test for the parent; the corrected version is not redundant, because R itself satisfies the parent condition but only witnesses the oracle condition when Oracle_R(t)=fail, which weak catching does not guarantee. The same typo in Definition 14 alters the characterization of weak functionality catching tests. These are substantive formal errors in a paper that presents formal definitions as a primary contribution.
  3. [3.2 and Section 6] All of the paper's definitions of hardening, catching, precision, and recall quantify over Oracle_R(t), and the paper assumes 'that, given a revision and a test, there is agreement on the oracle' (Section 3.2). The Catching JiTTest Challenge is therefore well-posed only relative to a chosen oracle; the challenge's notions of 'buggy' and 'false positive' inherit this dependence. The suggested tie-breaker (confining the decision to the pull request author, Section 3.2) is problematic because the goal is to catch bugs the author missed; the author's oracle at submission time may classify a genuinely catching test as a false positive. The paper should either state the well-posedness claim as conditional on an agreed oracle, or extend the framework to handle oracle disagreement, for example by defining precision and recall with respect to a set of oracles or a distribution over them.
minor comments (6)
  1. [Definition 2] The word 'Defintion' should be 'Definition'.
  2. [Example 4] The title contains 'pull reqest', which should be 'pull request'.
  3. [5.1] The sentence 'This is a good example the principal that' should read 'This is a good example of the principle that'.
  4. [Table 2] The table contains the typos 'categoiries' and 'whther', which should be 'categories' and 'whether'.
  5. [3.3] The tool name 'EvoStuite' should be 'EvoSuite'.
  6. [3.5] The notation R@P=p is used without an explicit formulaic definition; please define it precisely in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's formal definitions and the Catching JiTTest Challenge are self-contained, and its self-citations are motivational rather than load-bearing.

full rationale

The paper's derivation chain is not circular. The formal core (Definitions 2-15) is built from primitives such as Builds_R(t), Passes_R(t), Fails_R(t), and Oracle_R(t); hardening, catching, precision, and recall are then defined in terms of these primitives. The Catching JiTTest Challenge in Section 6 is a stated research objective, not a quantity derived from a fitted parameter or from a prior result. The delete-and-reinsert argument in Section 6.5 that any Catching JiTTest solution can find latent bugs is a direct logical corollary of Definition 11 and the construction, and the paper explicitly acknowledges the hard part there: 'The challenge of finding latent bugs with JiTTests lies primarily in locating additional sources of oracle information that help to determine what a re-inserted method (or code fragment) m is supposed to do.' The main assumptions are explicitly flagged rather than smuggled in: Section 3.2 says 'we assume that, given a revision and a test, there is agreement on the oracle for the expected behaviour of the test on that revision.' This is a stated conditionality, not a consequence of the definitions. The paper also flags a potential circularity in a future solution approach in Section 6.1: 'we would need to be cautious to ensure that our LLM-based test generation techniques do not become victims to the obvious circularity involved by also using LLMs to infer the code summaries.' That caution concerns a proposed technique, not the paper's own derivation. Self-citations such as Definition 1 ('taken from ref [8]') and the ACH/TestGen-LLM descriptions are contextual and motivational; the formal challenge does not depend on their validity. Consequently, there is no specific reduction of a claimed result to its own inputs, and no fitted input is relabeled as a prediction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 3 invented entities

No free parameters are fitted to data; the formal framework is self-contained. The main load-bearing assumptions are idealizations: independent test execution, tree-shaped revision history, and oracle agreement. The paper introduces named concepts (Catching JiTTest Challenge, Oracle Scavenging, Regression Only Trap) that are not empirically validated here.

free parameters (1)
  • precision threshold p = 0.8 (proposed, not fitted)
    Hand-selected threshold for R@P to define success for JiTTest generation; proposed as conservative choice in Section 3.5, not derived from data.
assumptions (4)
  • domain assumption Tests can be executed independently, so adding tests monotonically improves a suite's coverage and fault-revealing ability.
    Invoked in Definition 2 and used in Section 2.3 to argue Assured LLMST is easier than Assured LLMSE.
  • domain assumption Each revision has a unique parent in a tree repository.
    Definition 2 assumes a tree repository with Parent(R), used throughout the definitions and tables.
  • domain assumption For every revision and test, engineers agree on an oracle outcome.
    Section 3.2 assumes agreement; without it, the strong/weak and precision/recall definitions are not well defined.
  • domain assumption Future revisions exist over which hardening can be checked.
    Definitions 6-9 quantify over possible future revisions; this is inherently non-constructive but standard in testing theory.
invented entities (3)
  • Catching JiTTest Challenge
    purpose: Newly named research problem: generate tests just-in-time to catch bugs in a pull request before it ships.
    Central challenge of the paper; no empirical evaluation in this paper, so no independent falsifiable handle beyond the definitions themselves.
  • Oracle Scavenging
    purpose: Technique idea: extract expected behavior from comments and non-executable text to build oracles for LLM test generation.
    Illustrated by two hypothetical examples, not implemented or measured here.
  • Regression Only Trap
    purpose: Describes the limitation of regression-oracle-based test generation that cannot catch new bugs in the current revision.
    A framing concept; not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges." pith.science (2026). https://pith.science/paper/N6YZFQUZ

@misc{pith2026250416472,
  author       = {Pith},
  title        = {Pith review of: Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N6YZFQUZ}},
  note         = {Machine review of arXiv:2504.16472}
}
read the original abstract

Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching 'Just-in-Time' (JiTTest) Challenge, in which tests are generated 'just-in-time' to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025. Author order is alphabetical. The corresponding author is Mark Harman.

Figures

Figures reproduced from arXiv: 2504.16472 by the authors.

Figure 1
Figure 1. Hardening and catching Venn Diagram: Strong is a [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. YATE: The Role of Test Repair in LLM-Based Unit Test Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A test-repair pipeline, combining static analysis and re-prompting, raises LLM-generated unit test coverage and mutation killing by roughly 20-30 percent over a plain prompt baseline on six Java projects.

Reference graph

Works this paper leans on

90 extracted references · 47 canonical work pages · cited by 1 Pith paper

  1. [1]

    John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Ralf Laemmel, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers. 2020. WES: Agent-based User Interaction Simulation on Real Infrastructure (keynote paper). In 8𝑡ℎ Genetic improvement workshop (GI at ICSE 2020), Shin Yoo, ...

    2020

  2. [2]

    John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Erik Meijer, Silvia Sapora, and Justin Spahr-Summers. 2021. Testing Web Enabled Simulation at Scale Using Metamorphic Testing. In International Conference on Software Engineering (ICSE) Software Engineering in ...

    2021

  3. [3]

    Zhang, and Norm Zhou

    John Ahlgren, Kinga Bojarczuk, Sophia Drossopoulou, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Simon Lucas, Erik Meijer, Steve Omohundro, Rubmary Rojas, Silvia Sapora, Jie M. Zhang, and Norm Zhou. 2021. Facebook’s Cyber–Cyber and Cyber–Physical Digital Twins (keynote paper). In 25th International Conference on Evaluation ...

    2021

  4. [4]

    Nadia Alshahwan, Arianna Blasi, Kinga Bojarczuk, Andrea Ciancone, Natalija Gucevska, Mark Harman, Michal Krolikowski, Rubmary Rojas, Dragos Martac, Simon Schellaert, et al. 2024. Enhancing Testing at Meta with Rich-State Simu- lated Populations. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice . 1–12

    2024

  5. [5]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Mark Harman, Alexan- dru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using Large Language Models at Meta. In ACM International Con- ference on the Foundations of Software Engineering (FSE 2024) (Porto de Galinhas, Brazil, Brazil)

    2024

  6. [6]

    Nadia Alshahwan, Xinbo Gao, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, Taijin Tei, and Ilya Zorin. 2018. Deploying Search Based Software Engineering with Sapienz at Facebook (keynote paper). In 10𝑡ℎ International Symposium on Search Based Software Engineering (SSBSE 2018) . Montpellier, France, 3–45. Springer LNCS 11036

    2018

  7. [7]

    Nadia Alshahwan, Mark Harman, and Alexandru Marginean. 2023. Software Testing Research Challenges: An Industrial Perspective (keynote paper). In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST 2023) . IEEE, 1–10

    2023

  8. [8]

    Nadia Alshahwan, Mark Harman, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Assured LLM-Based Software Engineering (keynote paper). In 2𝑛𝑑. ICSE workshop on Interoperability and Robustness of Neural Software Engineering (InteNSE) (Lisbon, Portugal)

    2024

Show all 90 references
  1. [9]

    Nadia Alshahwan, Mark Harman, Alexandru Marginean, and Eddy Wang. 2024. Observation-based unit test generation at Meta. In Foundations of Software Engi- neering (FSE 2024)

    2024

  2. [10]

    Source Code Analysis and Manipulation Workshop (SCAM). 2001. SCAM Char- ter. https://www.ieee-scam.org/ The cited quotation has remained part of the workshop’s credo since its foundation in 2001

    2001

  3. [11]

    Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Jenny Li, Phil McMinn, and Hong Zhu

    Saswat Anand, Antonia Bertolino, Edmund Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Jenny Li, Phil McMinn, and Hong Zhu. 2013. An orchestrated survey of methodologies for automated software test case generation. Journa...

    2013

  4. [12]

    Kelly Androutsopoulos, David Clark, Haitao Dan, Mark Harman, and Robert Hierons. 2014. An Analysis of the Relationship between Conditional Entropy and Failed Error Propagation in Software Testing. In 36𝑡ℎ International Conference on Software Engineering (ICSE 2014) . Hyderabad...

    2014

  5. [13]

    Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

    Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

  6. [14]

    Carolin Brandt and Andy Zaidman. 2022. Developer-centric test amplification: The interplay between automatic generation human exploration. Empirical Software Engineering 27, 4 (2022), 96

    2022

  7. [15]

    Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209–224

    2008

  8. [16]

    Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser

    Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S. Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: preliminary assessment. In 33𝑟𝑑 International Conference on Software Engineering (ICSE’11) (Wa...

    2011

  9. [17]

    Cristian Cadar and Koushik Sen. 2013. Symbolic Execution for Software Testing: Three Decades Later. Commun. ACM 56, 2 (Feb. 2013), 82–90

    2013

  10. [18]

    Calcagno, D

    C. Calcagno, D. Distefano, J. Dubreil, D. Gabi, P. Hooimeijer, M. Luca, P. W. O’Hearn, I. Papakonstantinou, J. Purbrick, and D. Rodriguez. 2015. Moving Fast with Software Verification. In NASA Formal Methods - 7th International Symposium. 3–11

    2015

  11. [19]

    Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman

  12. [20]

    Maxime Cordy, Renaud Rwemalika, Adriano Franci, Mike Papadakis, and Mark Harman. 2022. FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . ACM...

    2022

  13. [21]

    De Millo, Richard J

    Richard A. De Millo, Richard J. Lipton, and Alan J. Perlis. 1979. Social Processes and Proofs of Theorems and Programs. Commun. ACM 22, 5 (May 1979), 271–280. An earlier version appeared in ACM Symposium on Principles of Programming Languages (POPL) , Los Angeles, California ,...

    1979

  14. [22]

    Dijkstra

    Edsger W. Dijkstra. 1978. On a Political Pamphlet from the Middle Ages (A response to the paper ‘Social Processes and Proofs of Theorems and Programs’ by DeMillo, Lipton, and Perlis). ACM SIGSOFT, Software Engineering Notes 3, 2 (1978), 14–17

    1978

  15. [23]

    Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W O’Hearn

  16. [24]

    Rajendra T Dodhiawala, NS Sridharan, Peter Raulefs, and Cynthia Pickering. 1989. Real-Time AI Systems: A Definition and An Architecture.. In IJCAI. Citeseer, 256–264

    1989

  17. [25]

    Elbaum, Hui Nee Chin, Matthew B

    Sebastian G. Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Matthew Jorde. 2009. Carving and Replaying Differential Unit Test Cases from System Test Cases.IEEE Transactions on Software Engineering 35, 1 (2009), 29–45

    2009

  18. [26]

    Angela Fan, Beliz Gokkaya, Mitya Lyubarskiy, Mark Harman, Shubho Sengupta, Shin Yoo, and Jie Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In ICSE Future of Software Engineering (FoSE 2023)

    2023

  19. [27]

    Zhiyu Fan, Haifeng Ruan, Sergey Mechtaev, and Abhik Roychoudhury. 2024. Oracle-guided Program Selection from Large Language Models. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 628–640

    2024

  20. [28]

    Jean Claude Fernandez, Claude Jard, Thierry Jéron, and César Viho. 1996. Using on-the-fly verification techniques for the generation of test suites. In Computer Aided Verification: 8th International Conference, CA V’96 New Brunswick, NJ, USA, July 31–August 3, 1996 Proceedings...

    1996

  21. [29]

    Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Hervé Robert, and Shubho Sengupta. 2025. Mutation-Guided LLM-based Test Generation at Meta. In 2025 ACM Conference on Foundations of Software Engineering (FSE 2025). ACM. Also available as ...

    2025 arXiv

  22. [30]

    Frankl, Stewart N

    Phyllis G. Frankl, Stewart N. Weiss, and Cang Hu. 1997. All-Uses vs Mutation Test- ing: An Experimental Comparison of Effectiveness. Journal of Systems Software 38 (1997), 235–253

    1997

  23. [31]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In 8𝑡ℎ European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE ’11). ACM, 416–419

    2011

  24. [32]

    Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed auto- mated random testing. In Programming Language Design and Implementation (PLDI 2005), Vivek Sarkar and Mary W. Hall (Eds.). ACM, 213–223

    2005

  25. [33]

    Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th international conference on software engineering. 72–82

    2014

  26. [34]

    Mark Harman. 2010. Why Source Code Analysis and Manipulation Will Always Be Important (Keynote Paper). In 10𝑡ℎ IEEE International Working Conference on Source Code Analysis and Manipulation . Timisoara, Romania

    2010

  27. [35]

    Mark Harman, Yue Jia, and William B. Langdon. 2011. Strong Higher Order Mutation-Based Test Data Generation. In 8𝑡ℎ European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE ’11) (Szeged, Hungary). ACM, New York...

    2011

  28. [36]

    Mark Harman, Yue Jia, and Yuanyuan Zhang. 2015. Achievements, open problems and challenges for search based software testing (keynote Paper). In 8𝑡ℎ IEEE International Conference on Software Testing, Verification and Validation (ICST 2015). Graz, Austria

    2015

  29. [37]

    Mark Harman and Bryan F. Jones. 2001. Search Based Software Engineering. Information and Software Technology 43, 14 (Dec. 2001), 833–839

    2001

  30. [38]

    Mark Harman, Afshin Mansouri, and Yuanyuan Zhang. 2012. Search Based Software Engineering: Trends, Techniques and Applications. Comput. Surveys 45, 1 (November 2012), 11:1–11:61

    2012

  31. [39]

    Mark Harman and Peter O’Hearn. 2018. From Start-ups to Scale-ups: Opportu- nities and Open Problems for Static and Dynamic Program Analysis (keynote paper). In 18𝑡ℎ IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2018) . Madrid, Spain, 1–23

    2018

  32. [40]

    Mark Harman and Laurence Tratt. 2007. Pareto Optimal Search-based Refactor- ing at the Design Level. In 9𝑡ℎ annual conference on Genetic and evolutionary computation (GECCO 2007). ACM Press, London, UK, 1106 – 1113. Harden and Catch for Just-in-Time Assured LLM-Based Software ...

    2007

  33. [41]

    Mark Harman, Xiangjuan Yao, and Yue Jia. 2014. A Study of Equivalent and Stubborn Mutation Operators Using Human Analysis of Equivalence. In 36𝑡ℎ International Conference on Software Engineering (ICSE 2014) . Hyderabad, India, 919–930

    2014

  34. [42]

    Kazi Amit Hasan, Marcos Macedo, Yuan Tian, Bram Adams, and Steven Ding

  35. [43]

    Rob Hierons, Kirill Bogdanov, Jonathan Bowen, Rance Cleaveland, John Derrick, Jeremy Dick, Marian Gheorghe, Mark Harman, Kalpesh Kapoor, Paul Krause, Gerald Luettgen, Tony Simons, Sergiy Vilkomir, Martin Woodward, and Hussein Zedan. 2009. Using Formal Methods to Support Testin...

    2009

  36. [44]

    Charles Anthony Richard Hoare. 1996. How did software get so reliable without proof?. In FME ’96: Industrial Benefit and Advances in Formal Methods: Third International Symposium of Formal Methods Europe (LNCS, 1051) . Springer-Verlag, 1–17

    1996

  37. [45]

    Charles Anthony Richard Hoare. 1996. How did software get so reliable without proof?. In IEEE International Conference on Software Engineering (ICSE’96) . IEEE Computer Society Press, Los Alamitos, California, USA. Keynote talk and extended abstract

    1996

  38. [46]

    Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. 2021. Online learning: A comprehensive survey. Neurocomputing 459 (2021), 249–289

    2021

  39. [47]

    Soneya Binta Hossain and Matthew Dwyer. 2024. TOGLL: Correct and strong test oracle generation with LLMs. arXiv preprint arXiv:2405.03786 (2024)

    2024 arXiv

  40. [48]

    Eugene Ie, Chih-wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019. RecSim: A Configurable Simulation Platform for Recommender Systems. arXiv e-prints, Article arXiv:1909.04847 (Sep 2019). arXiv:1909.04847 [cs.LG]

    2019 arXiv

  41. [49]

    Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering. 435–445

    2014

  42. [50]

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2024. TestGenEval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752 (2024)

    2024 arXiv

  43. [51]

    Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (September– October 2011), 649 – 678

    2011

  44. [52]

    Ernst, Reid Holmes, and Gordon Fraser

    René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing? Technical Report UW-CSE-14-02-02. University of Washington

    2014

  45. [53]

    Donald E. Knuth. 1984. Literate Programming. Comput. J. 27, 2 (1984), 97–111

    1984

  46. [54]

    Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv preprint arXiv:2410.21136 (2024)

    2024 arXiv

  47. [55]

    Hermann Kopetz and Wilfried Steiner. 2022. Real-Time Communication. In Real- time systems: Design principles for distributed embedded applications . Springer, 177–200

    2022

  48. [56]

    Kiran Lakhotia, Mark Harman, and Hamilton Gross. 2013. AUSTIN: An Open Source Tool for Search Based Software Testing of C Programs. Journal of Infor- mation and Software Technology 55, 1 (January 2013), 112–125

    2013

  49. [57]

    Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In 22𝑛𝑑 International Symposium on Foundations of Software Engineering (FSE 2014) , Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne Storey (Eds.). ACM, Hong Kong, Ch...

    2014

  50. [58]

    Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2013. Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Transactions on Software Engineering 40, 1 (2013), 23–42

    2013

  51. [59]

    Valentin J. M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2018. The Art, Science, and Engineering of Fuzzing: A Survey. CoRR abs/1812.00140 (2018). arXiv:1812.00140

    2018 arXiv

  52. [60]

    Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. In International Symposium on Software Testing and Analysis (ISSTA 2016). 94–105

    2016

  53. [61]

    Ke Mao, Timotej Kapus, Lambros Petrou, Ákos Hajdu, Matteo Marescotti, Andreas Löscher, Mark Harman, and Dino Distefano. 2022. FAUSTA: Scaling Dynamic Analysis with Traffic Generation at WhatsApp. In 15th IEEE Conference on Soft- ware Testing, Verification and Validation, ICST ...

    2022

  54. [62]

    James McGill. 2025. Time to First Review. https://docs.velocity.codeclimate. com/en/articles/2913584-time-to-first-review

    2025

  55. [63]

    Phil McMinn. 2004. Search-based Software Test Data Generation: A Survey. Software Testing, Verification and Reliability 14, 2 (June 2004), 105–156

    2004

  56. [64]

    Facundo Molina, Alessandra Gorla, and Marcelo d’Amorim. 2024. Test Oracle Automation in the era of LLMs. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  57. [65]

    Morten Mossige, Arnaud Gotlieb, and Hein Meling. 2015. Testing robot controllers using constraint programming and continuous integration. Information and Software Technology 57 (2015), 169–185

    2015

  58. [66]

    Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, and Peter C Rigby. 2024. CodeCompose: A Large-Scale Industrial Deployment of AI-assisted Code Authoring. In Foundations of Software Engineeri...

    2024 arXiv

  59. [67]

    Peter W O’Hearn. 2019. Incorrectness logic. Proceedings of the ACM on Program- ming Languages 4, POPL (2019), 1–32

    2019

  60. [68]

    Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Com- piler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 37𝑡ℎ International Conference on Software Engineering (ICSE 2015) . Florence, ...

    2015

  61. [69]

    Haraldsson, Mark Harman, William B

    Justyna Petke, Saemundur O. Haraldsson, Mark Harman, William B. Langdon, David R. White, and John R. Woodward. 2018. Genetic Improvement of Software: a Comprehensive Survey. IEEE Transactions on Evolutionary Computation 22, 3 (June 2018), 415–432

    2018

  62. [70]

    Juan Altmayer Pizzorno and Emery D Berger. 2024. Coverup: Coverage-guided LLM-based test generation. arXiv preprint arXiv:2403.16218 (2024)

    2024 arXiv

  63. [71]

    Outi Räihä. 2010. A survey on Search–Based Software Design. Computer Science Review 4, 4 (2010), 203–249

    2010

  64. [72]

    Aurora Ramirez, Jose Raul Romero, and Christopher L Simons. 2018. A systematic review of interaction in search-based software engineering. IEEE Transactions on Software Engineering 45, 8 (2018), 760–781

    2018

  65. [73]

    Aurora Ramirez, José Raúl Romero, and Sebastian Ventura. 2019. A survey of many-objective optimisation in search-based software engineering. Journal of Systems and Software 149 (2019), 382–395

    2019

  66. [74]

    David Schuler and Andreas Zeller. 2009. Javalanche: efficient mutation testing for Java. In 7𝑡ℎ joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE 2009). 297–298

    2009

  67. [75]

    Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. In 10𝑡ℎ European Software Engineering Conference and 13th ACM International Symposium on Foundations of Software Engineering (ESEC/FSE ’05) , Michel Wermelinger and Harald Gall (Eds.). ...

    2005

  68. [76]

    Simons, Ian C

    Christopher L. Simons, Ian C. Parmee, and Rhys Gwynllyw. 2010. Interactive, Evo- lutionary Search in Upstream Object-Oriented Class Design. IEEE Transactions on Software Engineering 36, 6 (2010), 798–816

    2010

  69. [77]

    John Steven, Pravir Chandra, Bob Fleck, and Andy Podgurski. 2000. jRapture: A capture/replay tool for observation-based testing. In Proceedings of the 2000 ACM SIGSOFT international symposium on Software Testing and Analysis (ISSTA 2000) . 158–167

    2000

  70. [78]

    Deepika Tiwari, Martin Monperrus, and Benoit Baudry. 2023. Mimicking Pro- duction Behavior with Generated Mocks. arXiv:2208.01321 [cs.SE]

    2023 arXiv

  71. [79]

    Shreshth Tuli, Kinga Bojarczuk, Natalija Gucevska, Mark Harman, Xiao-Yu Wang, and Graham Wright. 2023. Simulation-Driven Automated End-to-End Test and Oracle Inference. In 45th IEEE/ACM International Conference on Software Engi- neering: Software Engineering in Practice, SEIP@...

    2023

  72. [80]

    Lars van Hijfte and Ana Oprescu. 2021. Mutantbench: an equivalent mutant problem comparison framework. In2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 7–12

    2021

  73. [81]

    Margus Veanes, Colin Campbell, Wolfram Schulte, Pushmeet Kohli, N Tillmann, and W Grieskamp. 2005. On-the-fly testing of reactive systems. Submitted for publication (2005)

    2005

  74. [82]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2023. Software testing with large language model: Survey, landscape, and vision. arXiv preprint arXiv:2307.07221 (2023)

    2023 arXiv

  75. [83]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering . 1258–1268

    2024

  76. [84]

    Accessed March 27th 2025

    Michal Zalewski. Accessed March 27th 2025. American fuzzy lop. http://lcamtuf. coredump.cx/afl/

    2025

  77. [85]

    Shuyin Zhao. 2023. GitHub Copilot now has a better AI model and new capa- bilities. https://github.blog/2023-02-14-github-copilot-now-has-a-better-ai- model-and-new-capabilities/

    2023

  78. [86]

    Yuxiang Zhu and Minxue Pan. 2019. Automatic code summarization: A systematic literature review. arXiv preprint arXiv:1909.04352 (2019)

    2019 arXiv

  79. [2015]

    IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525

    The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525

    2015

  80. [2017]

    In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017

    An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 . 597–608

    2017

  81. [2019]

    Scaling static analyses at Facebook. Commun. ACM 62, 8 (2019), 62–70

    2019

  82. [2023]

    In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)

    Understanding the time to first response in GitHub pull requests. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 1–11

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.