{"id":"8d0594ab-245b-4243-bc5b-b9bf02e09d6f","arxiv_id":"2504.21161","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"JUDOT generates unit tests by using formal contracts extracted from Javadoc comments as search-based fitness functions, and reports testing 14.8% more documented behaviors and finding 45 real bugs in 118 Java classes.","lead":"This paper introduces JUDOT, a tool that turns Javadoc comments into search objectives for automatic unit test generation, and reports that it tests 14.8% more documented behaviors than a coverage-driven baseline. A reader might care because automated testing of documented behaviors is a long-standing weak spot in software engineering, and the tool also found 45 real bugs in open-source Java libraries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 14.8% tested-contract gain rests on JDOCTOR formula fidelity; the paper audits only failing contracts, and 58% of those are false alarms, so passing and uniquely hit contracts may not reflect documented behaviors.","rationale":"The reader's weakest assumption identifies exactly the load-bearing premise: JDOCTOR's inferred formulas must faithfully represent the natural-language Javadoc for the 14.8% tested-contract claim to mean what it says. The paper itself provides strong evidence that this premise is insecure: Section IV-F reports that 62 of 107 contract violations (58%) were false alarms due to imprecise inference, and Section III-E explicitly acknowledges that JUDOT depends on JDOCTOR's possibly imprecise translations. Importantly, the manual audit in RQ2 covers only failing contracts, not the 588 contracts counted as \"tested\" in RQ1. A contract can be counted as tested while its guard is a misinterpretation of the documented condition and the test passes; such a hit contributes to the headline increase without actually exercising the documented behavior. The 45 manually confirmed failures and the developer feedback are real and independently valuable, and they support the weaker claim that JUDOT can find genuine bugs. However, they do not repair the RQ1 metric, because the false-alarm rate on violations suggests that JDOCTOR formula fidelity is far from perfect, and the paper provides no fidelity measurement on the passing or uniquely hit contracts. This is not a fatal flaw: a bounded manual audit of the uniquely hit and sampled contracts would settle whether the gain is genuine. For that reason the appropriate verdict remains conditional rather than rejection, but the condition is substantive and should be stated explicitly in the final version.","tokens_in":18710,"tokens_out":6157,"duration_ms":67221,"concrete_test":"Take all 87 contracts uniquely tested by JUDOT plus a random sample of 50 contracts from the pass-both category; for each, independently translate the original Javadoc into a gold-standard contract, or have two annotators judge whether JDOCTOR's formula matches the Javadoc, and recompute the tested-contract counts using only formulas judged accurate. If precision on these samples is near 100%, the concern is resolved; if it approaches the 58% false-alarm rate observed on violations, the headline 14.8% gain would need substantial revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"JUDOT's central empirical claim is measured on JDOCTOR's inferred contracts, not on the natural-language Javadoc. A contract is counted as \"tested\" whenever a generated test satisfies JDOCTOR's precondition and guard formulas; the oracle and test name are then attached to that formula. The paper's own Section IV-F shows that when a violation is found, 62/107 (58%) are false alarms caused by imprecise natural-language inference. Those 107 are exactly the cases where the formula's assertion was exercised and failed; for the 588 \"tested\" contracts, including the 87 uniquely hit by JUDOT, no manual audit of formula fidelity is reported. If JDOCTOR's imprecision is systematic rather than concentrated in violation-inducing formulas, the headline 14.8% gain (87/588) may count contracts whose guards or assertions do not correspond to behaviors developers actually documented. The 45 actual failures and developer confirmations are valuable evidence for RQ2, but they do not validate the RQ1 metric: a passing hit on an incorrect contract is still not a documented behavior under test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JUDOT, a search-based test generator that treats Javadoc-derived contracts as first-class fitness functions rather than as post-hoc oracles. JUDOT uses JDOCTOR to infer precondition and postcondition contracts, defines distance-based objective functions for satisfying and for violating each postcondition contract, uses EvoSuite as the search backend with these contract-specific fitness functions, and post-processes generated tests to give them contract-derived names and oracles. The evaluation on 118 Java classes and 775 postcondition contracts compares JUDOT against EvoSuite+JDOCTOR, reporting 588 tested contracts (76.6%), 87 contracts uniquely tested by JUDOT, 31 additional contract violations, and, after manual inspection, 45 actual failures, with developer feedback on a subset. The central claim is that documentation-driven objectives cover behaviors that coverage-driven generators miss.","tokens_in":18880,"tokens_out":5790,"duration_ms":63601,"significance":"The core idea is timely and practically relevant: it moves documentation from being only a source of oracles to being the driver of the search, and it produces tests that are contextualized with meaningful names and assertions. The paper includes a replication package, manual inspection of all 107 reported violations, developer feedback on failure reports, and a clear motivating example. If the central claim holds, the 14.8% relative increase in tested contracts and the 45 confirmed failures are worthwhile results. However, the strength of the RQ1 claim is limited by the fact that the 'tested contract' metric is defined over JDOCTOR's inferred formulas, and the paper's own data show that these formulas are often imprecise for violation cases; this threat is acknowledged but not controlled for in the RQ1 measurement.","major_comments":[{"comment":"The RQ1 definition of a 'tested contract' counts a hit when a generated test executes the method with inputs satisfying the JDOCTOR guard and preconditions, and explicitly ignores the assertion. The paper's own Section IV-F reports that 62 of 107 contract violations (58%) were false alarms caused by imprecise natural-language inference. No manual audit or proxy validation is reported for the 588 tested contracts, including the 87 contracts uniquely hit by JUDOT. If JDOCTOR's imprecision is systematic rather than concentrated in violation-inducing formulas, a passing hit on an incorrect contract does not demonstrate that the documented behavior is under test. Please report a manual audit of a sample of passing and uniquely hit contracts, or recompute the RQ1 headline on a subset of human-validated contracts.","section":"Section IV-E and IV-F, Tables II and III"},{"comment":"The statistical support for the headline 14.8% increase is thin. The 5-out-of-10 majority threshold for declaring a contract 'tested' is arbitrary, no confidence intervals or significance tests are reported, and 10 repetitions is a small sample for proportion estimates. Report confidence intervals for the hit rates (e.g., Wilson intervals) and a sensitivity analysis across stricter thresholds (e.g., 7/10 and 9/10) to show that the 14.8% difference is not an artifact of the chosen cutoff.","section":"Section IV-D and IV-E"},{"comment":"The treatment of @throws contracts is unclear and potentially undefined. The paper represents the NoMoreException contract as 'drawer.exceeds(limit) => retVal instanceof NoMoreException' even though unwrapAndSave returns boolean; if the method throws, there is no return value. The paper does not explain how the evaluator programs obtain a value for 'retVal' on exceptional exits, or how the fitness function handles thrown exceptions. Since @throws contracts are part of the postcondition set counted in Table I, this gap affects the core technique and should be clarified by describing the actual instrumentation semantics for exceptional termination.","section":"Section III-A and III-B"},{"comment":"The time-budget comparison may favor JUDOT independently of the contract-specific fitness functions. JUDOT splits generation into multiple independent searches, with batches of up to 10 contract objectives, while EvoSuite+JDOCTOR is given a single continuous budget with all coverage fitness functions. Search restarts and objective decomposition can improve results by themselves, so the observed difference may not isolate the effect of contract-based objectives. Please add an ablation in which the backend is run with the same batching and time allocation but with coverage fitness functions, or explicitly discuss and control this confound.","section":"Section IV-D"}],"minor_comments":[{"comment":"The author name appears as 'Luca Gugliemo' in the header and as 'Luca Guglielmo' in the affiliation block; please correct the spelling.","section":"Title page"},{"comment":"The sentence 'we report that a test generator tests the contract c, if it hits c in at least 5 out of 10 executions' should be phrased as an operational definition chosen by the authors, not as a natural reporting convention.","section":"Section IV-E"},{"comment":"The notation for the two distance functions, DISTANCE_c(t) and DISTANCE_c(t), appears without a clear visual distinction between the contract-satisfying and contract-violating variants; please use distinct subscripts or superscripts throughout Section III.","section":"Section III-B"},{"comment":"When the normalization operator is applied to a sum of distances that are already normalized to [0,1], the result val/(1+val) is smaller than any individual term in the sum; a one-sentence intuition of why this is desirable would help readers.","section":"Section III-B"},{"comment":"The caption of Figure 2 does not state that test0 and test1 are deliberately invalid because they violate the documented precondition on limit; adding this note would make the illustration less confusing.","section":"Section II-B, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the RQ1 metric is closely aligned with the fitness function by construction, and the contract-fidelity problem is audited only on failure cases. The paper is honest about this in Sections III-E and IV-H, but the headline number is not yet convincing without an audit of passing contracts. The 45 actual failures and the contextualized test generation are solid contributions, and the replication package is a strength. If the authors add the requested validation and statistical analysis, the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: JUDOT is a genuine, incremental extension of JDOCTOR. Instead of using Javadoc-derived contracts only as oracles on separately generated tests, it makes each postcondition contract a fitness function that Evosuite minimizes, and attaches contract-specific names and oracles. The GiftPack example does real work: it shows a documented behavior that coverage-driven Evosuite consistently misses and JUDOT reliably catches.\n\nCredit where it is due. The baseline choice is right; Evosuite+JDOCTOR isolates the contribution, since both arms share the same contracts and back-end. The paper discloses the JDOCTOR precision problem up front (Section III-E), ships a replication package, manually inspects all 107 violations, and collected developer feedback on 26, with 9 confirmed as real. The 45 manually-confirmed failures are solid evidence, independent of any metric debate.\n\nNow the soft spots, in proportion. The RQ1 headline — 14.8% more contracts tested — is measured on JDOCTOR's inferred formulas, and 'tested' means a generated test satisfies the guard and preconditions, which is exactly what JUDOT's fitness function minimizes. That is not fully circular: the baseline uses coverage fitness, so the comparison does show contract-directed search reaches contracts that coverage-directed search misses. But the headline is partly a tautology about optimizing for the thing you then measure. The more serious concern is the one the stress-test note presses: the manual audit covered only the 107 violating contracts, and 62 of those (58%) were false alarms from imprecise natural-language inference. For the 588 passing hits, including the 87 uniquely hit by JUDOT, there is no fidelity check. If JDOCTOR's mis-inferences concentrate in violation-inducing formulas — plausible, since violations often sit on edge cases like NaN — the RQ1 numbers could be roughly right. But the paper does not show that, so 'tested contracts' is not yet established as 'documented behaviors under test.' This is a validity threat, not a fatal flaw: the authors flag the precision issue themselves in III-E and IV-F, and RQ2 is explicitly framed as the check on whether contracts match the Javadoc.\n\nMinor issues: the 5-of-10 majority hit rule is arbitrary, and no confidence intervals or statistical tests are reported, so the precision of the 14.8% is unknown. The time budget formula (35 seconds per fitness function, batches of 10) has free parameters that could tilt the comparison.\n\nWho this is for: researchers in search-based testing, test oracles, and natural-language-to-specification. It deserves a serious referee. Recommended path: major revision asking for a sensitivity analysis on the hit threshold and budget formula, confidence intervals on the headline numbers, and ideally a sample audit of passing hits or a breakdown of false alarms by contract shape.","headline":"JUDOT is a genuine, honest incremental contribution — Javadoc contracts as first-class search objectives — but the 14.8% headline is measured on inferred formulas, and the paper's own 58% false-alarm rate leaves the RQ1 metric under-validated.","tokens_in":19445,"tokens_out":5276,"would_cite":true,"duration_ms":49141,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating Javadoc contracts as test goals finds failures coverage-based tools miss.","keywords":["automated test generation","search-based testing","Javadoc","contract testing","test oracles","code comments","test objectives","Java"],"falsifier":"Take a fresh set of Java classes with human-annotated Javadoc, compute the inferred contract for every comment, and ask independent programmers whether the formula matches the prose; if a large share of formulas are judged wrong, then the 'tested contracts' metric measures agreement with the inference tool rather than with the documentation.","tokens_in":18461,"feed_emoji":"🧪","tokens_out":8918,"duration_ms":91170,"temperature":0.7,"pith_summary":"The paper aims to show that the behaviors programmers document in Javadoc comments are a usable, and mostly untapped, source of test objectives. It proposes a generator that converts each documented postcondition into a search objective, hunts for inputs that satisfy the contract's guard, and then tries both to satisfy and to violate the contract's assertion; each resulting test receives a name and oracle taken from the contract. Compared with a coverage-driven search-based generator that uses the same contracts only as oracles, the approach is reported to test 87 additional contracts out of 775 (a 14.8% increase) and to expose 31 contract violations the baseline missed. Manual inspection traced those violations to 45 actual program faults, which is the practical payoff: documentation-driven testing can find real bugs that structural coverage alone tends to skip.","feed_headline":"Mining Javadoc contracts tests 14.8% more behaviors","feed_subtitle":"A contract-focused search generator also surfaced 45 real program faults that code-coverage testing missed.","key_machinery":"The load-bearing object is the contract distance function. For a contract $c \\equiv \\bigwedge_j guard_j \\Rightarrow \\bigwedge_k assert_k$, JUDOT defines $DISTANCE_c(t) = (d^{in}_c(t)+d^{ret}_c(t))/2$, where $d^{in}_c$ accumulates normalized distances for satisfying all preconditions and guard terms, and $d^{ret}_c$ does the same for the assertion terms, with normalization $\\|v\\|=v/(1+v)$. The companion violation objective replaces the assertion sum by the product $\\prod_k d^{ret}_{\\neg assert_k}(t)$, so it reaches zero as soon as one conjunct of the assertion fails. These scalar functions, instantiated as evaluator programs, are fed to the search as fitness functions; the genetic search minimizes them and stops when the value is zero, producing tests that either satisfy or violate the focal contract. This machinery turns natural-language documentation into a smooth optimization landscape, and it is what lets the search set up situations that are behaviorally required but structurally incidental, such as a drawer that already contains the gift.","core_discovery":"JUDOT's central claim is that contract formulas inferred from Javadoc—preconditions on parameters and implication-style postconditions of the form guard ⇒ assertion—should be treated as first-class test objectives rather than as post-hoc oracles. For every postcondition contract the search builds two distance functions: one measures how close a candidate test is to satisfying the contract, the other how close it is to violating it; the genetic search minimizes both, and when a violating input is found that outcome is prioritized. The paper reports that on 118 Java classes this strategy steadily tests 586 of 775 postcondition contracts, including 87 contracts a coverage-driven baseline never hit, and reveals 107 contract violations, 31 more than the baseline. After manual triage, 45 of those violations correspond to actual failures of the classes under test, including defects developers later confirmed or fixed. The one-to-one mapping between tests and focal contracts also gives each test case a name derived from the Javadoc wording and an oracle from the contract assertion.","pith_inferences":["If contract-testing were adopted, the natural next step would be to define a coverage metric over contracts—percentage of inferred contracts hit—and use it to prioritize which Javadoc comments need human review when the inferred formula is imprecise.","The contract-as-objective framing should transfer to other documentation conventions or to runtime assertion checking, since the search only needs guard and assertion formulas, not Javadoc specifically.","The 45 confirmed failures likely understate the bug-finding value, because failures in discontinued or rewritten projects could not be reported to developers, and because multi-contract baseline tests can mask one contract failure with another.","A testable extension would be to rank classes by contract density and compare the faults found per hour by contract-driven testing against mutation testing, which would tell teams when documentation-based testing is worth the investment."],"forward_implications":["The number of documented behaviors exercised by a test suite can be measured contract-by-contract, giving maintainers a coverage-like metric tied to intent rather than to code shape.","Tests generated this way come with meaningful names and assertions tied to one contract, so a failure directly identifies which documented behavior broke.","Because the violating-objective search explicitly tries to break each contract, failures can be detected even when no developer has written an expected output in advance.","The technique is complementary to structural coverage: two contracts in the benchmark were tested only by the coverage-driven baseline, suggesting a combined fitness function could test more contracts than either approach alone.","The false-alarm cost is real: 62 of 107 reported violations were traced to imprecise natural-language inference, so the approach shifts some effort from test generation to contract triage."],"supporting_citations":[{"why":"Supplies the method that translates Javadoc text into the precondition and postcondition formulas JUDOT turns into search objectives.","marker":"[7]"},{"why":"Provides the search-based test generation engine used as JUDOT's backend and as the core of the coverage-driven baseline.","marker":"[25]"},{"why":"Defines the oracle problem that motivates deriving test oracles from contracts.","marker":"[8]"},{"why":"Provides the benchmark of 118 Java classes and the contract-extraction setup on which the experiments run.","marker":"[33]"},{"why":"Supplies the expected-output resources of the benchmark used to configure contract inference.","marker":"[34]"},{"why":"The natural-language parser used to derive test names from Javadoc wording, supporting the contextualization claim.","marker":"[26]"},{"why":"Provides the dependency-parsing method behind the name synthesis from guard and assertion descriptions.","marker":"[27]"}],"fun_headline_variants":["Javadoc contracts as test objectives find 45 real faults","Contract-based test generation from Javadoc finds 45 genuine bugs","Mining Javadoc for test objectives beats coverage-driven testing","JUDOT: Using Javadoc to target behaviors and reveal faults"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the contract formulas automatically inferred from Javadoc mean what the developers meant; the paper itself reports that 62 of 107 detected violations were false alarms traceable to imprecise natural-language inference, so the whole comparison inherits that imprecision.","fun_headline_variants_meta":{"raw":{"variants":["Javadoc contracts as test objectives find 45 real faults","Contract-based test generation from Javadoc finds 45 genuine bugs","Mining Javadoc for test objectives beats coverage-driven testing","JUDOT: Using Javadoc to target behaviors and reveal faults"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3353,"prompt_tokens":838,"completion_tokens":2515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":2441}},"tokens_in":454,"tokens_out":2515,"duration_ms":17574,"temperature":1.0,"reasoning_tokens":2441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:11:25.249141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of Java classes with human-annotated Javadoc, compute the inferred contract for every comment, and ask independent programmers whether the formula matches the prose; if a large share of formulas are judged wrong, then the 'tested contracts' metric measures agreement with the inference tool rather than with the documentation.","supporting_citations":[{"cited_title":"Translating code comments to procedure spec- ifications,","cited_arxiv_id":null,"evidence_quote":"Supplies the method that translates Javadoc text into the precondition and postcondition formulas JUDOT turns into search objectives."},{"cited_title":"Release ISSTA 2016 · albertogoffi/toradocu","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark of 118 Java classes and the contract-extraction setup on which the experiments run."},{"cited_title":"toradocu/src/test/resources/goal-output at master · albertogoffi/- toradocu · GitHub","cited_arxiv_id":null,"evidence_quote":"Supplies the expected-output resources of the benchmark used to configure contract inference."},{"cited_title":"The stanford corenlp natural language processing toolkit,","cited_arxiv_id":null,"evidence_quote":"The natural-language parser used to derive test names from Javadoc wording, supporting the contextualization claim."},{"cited_title":"Generating typed dependency parses from phrase structure parses","cited_arxiv_id":null,"evidence_quote":"Provides the dependency-parsing method behind the name synthesis from guard and assertion descriptions."}],"review_version":1}