{"id":"780321c2-29af-4676-8b4d-591de202a600","arxiv_id":"2504.16472","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"This paper formally defines hardening and catching tests, introduces the Catching Just-in-Time Test (JiTTest) Challenge for LLM-generated software tests, and argues it is the most impactful open problem in automated test generation.","lead":"This paper defines two kinds of software tests, hardening tests that protect against future regressions and catching tests that fail on a current code change, and it proposes a research challenge for generating catching tests just before a pull request ships. It is a keynote-style position paper from Meta authors intended to steer LLM-based test generation research toward just-in-time fault catching.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The formalization assumes an agreed oracle, but the Catching JiTTest Challenge's notions of 'buggy' and 'false positive' are oracle-relative; without a fixed oracle, the central claim of well-posedness is conditional.","rationale":"I agree with the Reader's verdict of CONDITIONAL and with the identification of oracle agreement as the weakest load-bearing assumption. The Catching JiTTest Challenge is the paper's primary contribution; its formal definition and the precision/recall framework (Section 3.5) are meaningful only if Oracle_R(t) is well-defined. The paper's own discussion (Section 3.2, Section 6.4) stresses that oracles are socially contested and may change over time, then assumes agreement for the sake of the formalism. This is a reasonable simplification for a position paper, but it means the 'well-posed' part of the central claim is conditional on an assumption that the paper does not justify. The formal defects the Reader mentions (e.g., the quantifier error in Definition 9, the typo in Definition 12) are real and should be corrected, but they are secondary to the oracle issue. The delete-and-reinsert argument is logically valid if a JiTTest generator exists for arbitrary PRs, but the oracle difficulty is most acute there. A CONDITIONAL verdict is appropriate: the paper makes a valuable conceptual contribution that is likely to stimulate research, but its formal foundations require a treatment of the oracle parameter before the central challenge can be regarded as fully well-posed. My concrete test would settle whether the concern actually lands by showing whether the formal results are invariant under oracle choice.","tokens_in":27973,"tokens_out":15756,"duration_ms":143296,"concrete_test":"Re-derive the formalization of Sections 3-6 with an explicit oracle parameter O: replace every occurrence of Oracle_R(t) with O_R(t) and restate the Catching JiTTest Challenge as a family of problems parameterized by O. Check whether the eight categories in Figure 1, the definitions of strong/weak catching and hardening, and the R@P=p objective remain invariant under different choices of O (e.g., PR author vs. a test engineer vs. a user). Then verify whether the Section 6.5 delete-and-reinsert implication holds for any O or only for O with special properties. If the classifications and the challenge's objective vary with O, the paper must either commit to a single operational oracle or explicitly relativize its claims; if they do not vary, the oracle-agreement concern is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the Catching JiTTest Challenge (Section 6) is well-posed, untackled, and impactful. Its formal definition ('automatically generate a test that can reveal bugs in the pull request, with low risk of false positive failure on non-buggy pull requests') and the preceding eight-way classification (Section 4) all quantify over Oracle_R(t) introduced in Definition 5. In Section 3.2 the authors explicitly recognize that software engineering is a partly social process and that 'one engineer's (and/or one user's) oracle may differ from another,' then assume 'that, given a revision and a test, there is agreement on the oracle.' This assumption is load-bearing: if two stakeholders disagree about whether a test should pass on a revision, the same generated test can be a strong catching test under one oracle and a false positive under another, and a pull request can be simultaneously 'buggy' and 'non-buggy' depending on the oracle. The paper's suggested tie-breaker, confining the decision to the pull request author, does not resolve the issue because the challenge explicitly aims to catch bugs the author may have missed—so the author's own oracle at submission time may classify the test as a false positive even when the test is correct. The delete-and-reinsert argument (Section 6.5) also relies on an oracle for the re-inserted method, which is especially hard to fix in the absence of agreed non-executable specification. Consequently, the well-posedness half of the central claim depends on a premise the paper itself flags as unreasonable in practice, and the formalization would need to either parameterize the oracle explicitly or provide a substantive account of how agreement is reached.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes two kinds of automated tests in a revision-tree setting: hardening tests, which pass on the current revision and are intended to catch future regressions, and catching tests, which fail on the current revision and may reveal bugs in it. It defines weak and strong variants of each, distinguishes regression-catching from functionality-catching tests, and presents an eight-way classification of test signals (Section 4). The main proposal is the Catching JiTTest Challenge (Section 6): given a buggy pull request and its parent, automatically generate a test that can reveal bugs in the pull request with low risk of false-positive failure on non-buggy pull requests. The paper also analyzes deployment options for timely and just-in-time tests (Section 5), and argues that a solution to the challenge can be repurposed to find latent bugs in legacy code by deleting and re-inserting code as a synthetic pull request (Section 6.5). The paper is written as an FSE 2025 keynote companion and draws on the authors' industrial experience with LLM-based test generation at Meta.","tokens_in":28306,"tokens_out":13828,"duration_ms":120542,"significance":"The paper's central conceptual contribution is to identify and name the Catching JiTTest Challenge, which is a plausible and practically important gap in automated test generation: most current techniques harden against future regressions, while catching bugs in the pull request being reviewed is comparatively under-addressed. The deployment tables (Tables 1-3) provide a useful decision-oriented taxonomy, and the delete-and-reinsert argument (Section 6.5) is an elegant reduction that broadens the challenge's impact. The paper is also appropriately candid about the oracle problem and the social nature of software engineering, and the 'oracle scavenging' idea is a promising research direction. However, the claimed 'Precise Formal Foundations' currently contain several errors: Definition 9 is not a well-formed formula, and Definitions 12 and 14 contain an apparently trivial existential condition. These must be corrected before the definitions can be used as stated. The well-posedness of the Catching JiTTest Challenge is conditional on the oracle-agreement assumption, which should be stated explicitly.","major_comments":[{"comment":"Definition 9, the definition of Perfect Recall Hardening Test, is not a well-formed formula: the first occurrence of R' in 'Oracle_R'(t)=fail' is free, while the second occurrence is bound by the universal quantifier in '∀R'.Fails_R'(t)'. The intended perfect-recall property is presumably Hardening_R(t) ∧ ∀R' . (Oracle_R'(t)=fail ⇒ Fails_R'(t)). As written, the definition cannot be used to derive the 'perfect precision and recall' conjunction, and this undermines the paper's claim of precise formal foundations.","section":"3.4"},{"comment":"Definitions 12 and 14 both contain the existential clause '∃R'.Parent(R)=Parent(R)', which is trivially true because Parent(R) is identical to itself. The intended condition is '∃R'.Parent(R')=Parent(R)'. In Definition 12 this error reduces a weak regression catching test to merely 'passes on parent and fails on R', dropping the requirement that the test be a hardening test for the parent; the corrected version is not redundant, because R itself satisfies the parent condition but only witnesses the oracle condition when Oracle_R(t)=fail, which weak catching does not guarantee. The same typo in Definition 14 alters the characterization of weak functionality catching tests. These are substantive formal errors in a paper that presents formal definitions as a primary contribution.","section":"3.7 and 3.8"},{"comment":"All of the paper's definitions of hardening, catching, precision, and recall quantify over Oracle_R(t), and the paper assumes 'that, given a revision and a test, there is agreement on the oracle' (Section 3.2). The Catching JiTTest Challenge is therefore well-posed only relative to a chosen oracle; the challenge's notions of 'buggy' and 'false positive' inherit this dependence. The suggested tie-breaker (confining the decision to the pull request author, Section 3.2) is problematic because the goal is to catch bugs the author missed; the author's oracle at submission time may classify a genuinely catching test as a false positive. The paper should either state the well-posedness claim as conditional on an agreed oracle, or extend the framework to handle oracle disagreement, for example by defining precision and recall with respect to a set of oracles or a distribution over them.","section":"3.2 and Section 6"}],"minor_comments":[{"comment":"The word 'Defintion' should be 'Definition'.","section":"Definition 2"},{"comment":"The title contains 'pull reqest', which should be 'pull request'.","section":"Example 4"},{"comment":"The sentence 'This is a good example the principal that' should read 'This is a good example of the principle that'.","section":"5.1"},{"comment":"The table contains the typos 'categoiries' and 'whther', which should be 'categories' and 'whether'.","section":"Table 2"},{"comment":"The tool name 'EvoStuite' should be 'EvoSuite'.","section":"3.3"},{"comment":"The notation R@P=p is used without an explicit formulaic definition; please define it precisely in the text.","section":"3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the FSE Companion keynote track well. The formal errors, while local, are in the paper's stated primary contribution, so I recommend a careful revision and re-verification of all definitions after the proposed corrections. The oracle-agreement issue should be addressed in the main text rather than only in the discussion. No other concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the Catching JiTTest Challenge is worth taking seriously: generating a test for a pull request that reveals a bug in that request, with low false-positive risk, is a crisp and currently undertheorized target. Second, the formal scaffolding around it is not yet trustworthy — several definitions have quantifier and equality errors, and the whole framework leans on an oracle-agreement assumption the authors themselves concede is an idealization. Don't desk-reject it, but don't treat the formalization as settled either.\n\nWhat's genuinely good: the hardening/catching split, the regression-catching vs functionality-catching distinction, and the eight-way classification of test signals all give the community a cleaner vocabulary. The deployment tables are unusually thoughtful, especially the observation that running generated tests on the parent revision yields extra signal for free. Naming R@P=p as the goal is also a real step forward, and the oracle-scavenging idea — mining PR titles, comments, and other non-executable text for intended behavior — points at a research direction LLM-based testing can actually chase.\n\nThe soft spots are real but mostly fixable. Definition 9 mis-scopes a quantifier; Definitions 12 and 13 contain a tautological equality and then drop the existential that the definition requires; Definition 15's simplification omits the disjunct covering the \"no sibling revision with a correct failure\" case. These are mechanical repairs, but they undermine the claim of \"precise formal foundations.\" The bigger conceptual gap is oracle agreement. The paper says oracles are socially contested, then assumes agreement. The suggested tie-breaker — let the PR author decide — doesn't rescue the challenge, because the challenge specifically targets bugs the author may have missed. So the well-posedness claim is conditional: without an agreed oracle, the same generated test can be a true catch or a false positive depending on whose oracle you adopt. That's not fatal for a position paper, but it should be parameterized explicitly rather than assumed away. The abstract also promises \"initial results,\" but those results live in other papers; this one is a challenge statement, and the abstract slightly overstates its contents.\n\nThis is a paper for automated-testing researchers, LLM-based SE folks, and industrial practitioners deciding where to point test-generation investment. It deserves serious refereeing: the challenge is impactful, and the taxonomy will shape how people talk about just-in-time testing. I'd send it to review, ask for a revision that fixes the formal errors and states the oracle-dependence up front, and be willing to accept once those are addressed.","headline":"The Catching JiTTest Challenge is a real and impactful research target, but the formal definitions and the oracle-agreement assumption need work before they can carry the weight this paper puts on them.","tokens_in":28838,"tokens_out":3322,"would_cite":false,"duration_ms":34046,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the next big open problem in automated test generation is producing tests that catch bugs in a pull request just before it lands, and that LLMs' ability to read code and natural language makes the problem tractable.","keywords":["just-in-time testing","hardening tests","catching tests","regression oracle","oracle scavenging","LLM-based test generation","pull request testing","precision and recall in testing"],"falsifier":"Take a set of real pull requests with known bugs and a set of clean ones from the same repository; run a JiTTest generator that sees only the pull request, its parent, and the pull request's own text; then measure whether its failing tests distinguish the two sets at the promised precision. If no generator can beat a random-or-trivial baseline without false positives, the premise that the challenge is ready to be solved collapses.","tokens_in":27804,"feed_emoji":"🧪","tokens_out":6247,"duration_ms":55669,"temperature":0.7,"pith_summary":"The paper aims to sharpen two fundamental notions in automated software testing: hardening tests, which pass on the current code and guard against future regressions, and catching tests, which fail on the current change and expose a bug in it. It argues that mainstream test generation has fallen into a 'Regression Only Trap' by relying on the regression oracle, so it cannot catch bugs in new functionality. The central open problem it formulates is the Catching JiTTest Challenge: given a buggy pull request and its parent revision, automatically generate a test that reveals the pull request's bug with low risk of failing on non-buggy pull requests. A sympathetic reader would care because the paper shows this challenge is well-posed, currently untackled, and that solving it would also yield a way to uncover latent bugs in legacy code by deleting and re-inserting code as a fake pull request. The paper grounds its optimism in the ability of large language models to scavenge intended behaviour from non-executable text such as pull request titles and comments.","feed_headline":"Catching pull-request bugs before they land: testing's open goal","feed_subtitle":"A formal framework separates hardening tests from catching tests and says LLMs can make the latter work.","key_machinery":"The load-bearing machinery is a family of definitions built on a single object: a partial oracle Oracle_R(t) that states whether test t is expected to pass or fail on revision R, together with the predicates Builds, Passes, and Fails. Hardening tests are defined through the regression oracle: they pass on the current revision and fail correctly on some child revision. Catching tests are defined by failure on the current revision, with strong catching requiring the oracle to call that failure correct. The eight-cell classification (weak/strong hardening on the parent, weak/strong catching on the child) is what organises the paper's deployment analysis: it maps each cell to a decision—land, report signal, or discard. The Just-in-Time definition is the other key object: a JiTTest for revision R is timely for R but not timely for its parent, which means it can be generated after the pull request appears but before first human review.","core_discovery":"On the paper's own terms, the discovery is a classification plus a challenge. Using an oracle that says whether a given test should pass or fail on a given revision, the paper defines weak and strong hardening tests (passing now and correctly failing on some future revision) and weak and strong catching tests (failing now, correctly or not). Crossing hardening on the parent with catching on the child yields eight behavioural categories, of which only three give reliable signals; five are misleading. The paper's headline claim is that the Catching JiTTest Challenge—generate, just before a pull request lands, a test that fails on that pull request when it is buggy and does not fail on non-buggy ones—is the most impactful currently open problem in automated test generation. It further claims that any solution can be repurposed: delete a code fragment, re-insert it as a pull request, generate a catching JiTTest, and the same machinery finds latent bugs already in production. The paper proposes measuring success by recall at fixed precision, R@P=p, rather than by aggregates that trade precision against recall.","pith_inferences":["If the JiTTest framework is adopted, precision-at-threshold (R@P=p) could replace F1-style aggregation as the default metric for industrial test generators, because it encodes the asymmetry that false positives are costlier than missed bugs.","Oracle scavenging from non-executable text suggests a testable benchmark: measure whether giving an LLM only a parsimonious pull request title and code comments yields higher catch rates than giving it the code alone.","The delete-and-reinsert construction implies that a mature codebase's history of reverts or bug-fix commits could be mined to create a labelled evaluation set for catching JiTTests, without needing newly injected faults.","The paper's oracle caveat points to a social extension: in contested behaviours, the JiTTest system could report the disagreement rather than a pass/fail verdict, turning the oracle friction into a requirements-elicitation signal."],"forward_implications":["A solution to the Catching JiTTest Challenge would let automated testing catch bugs in new functionality, not just regressions, breaking the Regression Only Trap.","Any such solution doubles as a latent-bug finder: delete-and-reinsert turns arbitrary legacy code into a fresh pull request for the generator to test.","Running a generated JiTTest on the parent revision is cheap and gives extra signal; the paper concludes this should become standard practice.","Failing JiTTests that also fail on the parent should usually be discarded unless new techniques can separate true bugs from false positives.","Human review of weak hardening tests turns them into strong hardening tests, removing three of the five misleading signal categories."],"supporting_citations":[{"why":"Supplies the definition of oracle and the Oracle Problem that the framework's oracle agreement assumption addresses.","marker":"[13]"},{"why":"Defines the regression oracle, the mechanism by which hardening tests judge future revisions.","marker":"[7]"},{"why":"Provides the generate-and-test assurance approach that the authors' LLM test generation and the online-versus-offline timing distinction build on.","marker":"[8]"},{"why":"Describes the deployed LLM test improvement system whose human-review step turns weak hardening tests into strong hardening tests.","marker":"[5]"},{"why":"Provides the mutation-guided generator that guarantees generated tests are hardening and kill uncaught mutants.","marker":"[29]"},{"why":"Explains failed error propagation, the mechanism by which latent bugs stay hidden and become the 'unexploded bombs' that JiTTest latent-bug detection targets.","marker":"[12]"},{"why":"Supports the claim that software engineering is a partly social process, justifying the paper's oracle-agreement assumption and its caveats.","marker":"[21]"},{"why":"Provides the roughly eight-hour first-review bound that makes just-in-time test generation computationally feasible.","marker":"[62]"}],"fun_headline_variants":["JIT test generation: the untapped goal in LLM testing","Catching bugs before landing: LLMs' next test frontier","Hardening vs catching: a new test taxonomy for LLMs","Just-in-time tests to catch pull-request bugs","R@P metric redefines LLM test generation success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole framework depends on engineers agreeing, for each revision and test, on whether the test's expected outcome is pass or fail; if one engineer calls a failure a bug and another calls it intended behaviour, the eight-way classification and the Catching JiTTest Challenge lose their precision.","fun_headline_variants_meta":{"raw":{"variants":["JIT test generation: the untapped goal in LLM testing","Catching bugs before landing: LLMs' next test frontier","Hardening vs catching: a new test taxonomy for LLMs","Just-in-time tests to catch pull-request bugs","R@P metric redefines LLM test generation success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000428,"raw_usage":{"total_tokens":2222,"prompt_tokens":1014,"completion_tokens":1208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":1123}},"tokens_in":630,"tokens_out":1208,"duration_ms":8649,"temperature":1.0,"reasoning_tokens":1123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:02:03.486792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real pull requests with known bugs and a set of clean ones from the same repository; run a JiTTest generator that sees only the pull request, its parent, and the pull request's own text; then measure whether its failing tests distinguish the two sets at the promised precision. If no generator can beat a random-or-trivial baseline without false positives, the premise that the challenge is ready to be solved collapses.","supporting_citations":[{"cited_title":"De Millo, Richard J","cited_arxiv_id":null,"evidence_quote":"Supports the claim that software engineering is a partly social process, justifying the paper's oracle-agreement assumption and its caveats."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the roughly eight-hour first-review bound that makes just-in-time test generation computationally feasible."}],"review_version":1}