{"id":"cb9a514b-81b1-4201-8499-8d8af63dad10","arxiv_id":"2607.04325","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Native Overlay support in EvoMaster lets testers supply named examples that improve black-box REST API fuzzing results on industrial systems without vendor-specific formats.","lead":"The authors added native OpenAPI Overlay support to the EvoMaster REST fuzzer so testers can inject named example values without editing schemas. Trials on five industrial APIs from companies of different sizes and countries showed higher 2xx coverage when overlays were used.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"The design confounds Overlay with hand-crafted examples; Table 1 gains cannot isolate the standard as the causal factor.","rationale":"The reader correctly flags the single short run and the engineer-written examples as the weakest assumption and therefore assigns CONDITIONAL. That diagnosis is right but incomplete: the deeper load-bearing gap is the missing control that would isolate Overlay-as-delivery-mechanism from examples-as-content. The paper’s own text (§1, §5) already concedes the functional equivalence, yet still attributes the Table 1 improvements to Overlay. Because the study is framed as a feasibility demonstration rather than a controlled causal experiment, and because the authors openly list the same limitations, the CONDITIONAL verdict remains appropriate; the concern simply sharpens the condition that must still be met (usability/maintainability evidence or a proper three-arm comparison) before the claim can be accepted unconditionally. No stronger rejection is warranted: the engineering contribution (native Overlay support, open-source overlay-jvm, named-example combination handling) is real and the industrial feedback is positive.","tokens_in":11653,"tokens_out":597,"duration_ms":7420,"concrete_test":"On any one of the five APIs (or a public proxy), re-run the identical 10-minute EvoMaster configuration three ways: (1) baseline schema, (2) schema after Overlay merge, (3) schema with the exact same named examples inserted by direct edit (no Overlay). If 2xx coverage and fault counts of (2) and (3) are statistically indistinguishable while both beat (1), the gains are due to examples, not Overlay; the viability claim for the standard then requires separate usability/maintainability evidence.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (abstract, §7) is that Overlay is a viable solution that better enables black-box REST fuzzing. The only quantitative support is Table 1: five single 10-minute EvoMaster runs with vs. without Overlay, showing higher 2xx (or success-code) coverage on A0–A3. As the authors themselves note (§1, §5), from the fuzzer’s perspective Overlay is equivalent to manually editing examples into a schema copy—the technique already shown useful in prior work [2]. The experiment therefore never separates (a) the value of domain-specific example values from (b) the value of delivering those values via the Overlay standard rather than via schema mutation or a proprietary seed file. Because the same engineers both wrote the Overlay files and ran the short sessions, any observed lift is most parsimoniously attributed to the quality of the hand-crafted examples, not to Overlay itself. Without a third arm that injects identical examples by a non-Overlay route, the claim that “Overlay is a viable solution” rests on an untested causal attribution.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes using the OAI Overlay standard to inject named 'examples' into OpenAPI schemas so that black-box REST API fuzzers can be steered with domain-specific test data without permanently editing the schema or inventing a proprietary seed format. The authors implement native Overlay support (including multi-parameter named-example combinations and archive retention) in EvoMaster via an open-source JVM library, and evaluate it on five industrial APIs from five enterprises of different sizes and geographies. Table 1 reports that 10-minute Overlay-augmented runs improve 2xx (or success-code) endpoint coverage relative to baseline EvoMaster on four of the five APIs, with practitioner feedback that Overlay is usable once learned. The central claim is that Overlay is a viable, maintainable industrial solution for supplying such hints.","tokens_in":11887,"tokens_out":950,"duration_ms":8885,"significance":"If the result holds, the work offers a practical, standards-based answer to a recurring industrial pain point: how to give black-box REST fuzzers domain knowledge without vendor lock-in or schema pollution. Strengths that should be credited include the open-source overlay-jvm library, native multi-example combination handling and web-report tracing inside EvoMaster, and a genuinely multi-enterprise industrial evaluation (Fortune-500 firms plus a three-person startup) that is rare in the REST-fuzzing literature. Even if the quantitative gains are modest and confounded, the engineering artifact and the demonstration that practitioners can write Overlay files for real APIs remain useful contributions to the tooling ecosystem.","major_comments":[{"comment":"§5 and Table 1: the only quantitative evidence is five single 10-minute with/without Overlay runs. There are no repeated trials, no variance estimates, and no statistical tests. Given that two APIs already saturate or are limited by unrelated fuzzer bugs (A3 PATCH media-type fault, A4 full coverage without Overlay), the directional improvements cannot be treated as robust evidence that Overlay itself improves fuzzing effectiveness.","section":"§5, Table 1"},{"comment":"Abstract, §1 and §5: the design confounds the Overlay delivery mechanism with the quality of the hand-crafted examples. The authors themselves note that, from the fuzzer’s perspective, Overlay is equivalent to manually editing examples into a schema copy (already shown useful in prior work [2]). Without a third arm that injects identical example values by a non-Overlay route (schema mutation or proprietary seed file), the claim that “Overlay is a viable solution” cannot isolate the standard as the causal factor; the observed lift is most parsimoniously attributed to the examples themselves.","section":"Abstract, §1, §5"}],"minor_comments":[{"comment":"Table 1 column headers (B-2xx / O-2xx) are never defined in the caption; a one-sentence clarification that they count endpoints returning at least one 2xx (or success code for A2) would help.","section":"Table 1"},{"comment":"§4: the probability P used for named-example combination sampling is mentioned only as “e.g., P = 0.5”; stating the actual default used in the experiments would improve reproducibility.","section":"§4"},{"comment":"Figure 4 is referenced but its content is not described in the text; a short caption explaining what the “Examples” view shows would make the usability claim more concrete.","section":"§4, Figure 4"},{"comment":"Several self-citations to concurrent or in-press EvoMaster papers ([2], [3], [25]) are used to justify baseline behaviour; ensuring that the essential claims are self-contained would help readers who cannot access those works.","section":"§2, §4"}],"recommendation":"major_revision","confidential_remarks":"The industrial multi-enterprise setting is a genuine strength and should be weighed heavily. The main risk is over-claiming: the paper essentially re-packages the already-known value of examples under a new standard. A revision that either adds a non-Overlay control arm or reframes the contribution strictly as “engineering feasibility and usability of Overlay for example injection” would make the manuscript much stronger for a journal audience."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean, modest industrial paper. The new piece is not “examples help fuzzers” (they already knew that from their own prior work) but the first native Overlay integration in a maintained fuzzer, plus a five-company study that shows practitioners can actually write the files and get directional coverage lifts.\n\nWhat they did well: they shipped overlay-jvm as a separate open-source library, extended EvoMaster’s sampling and archive so named-example combinations are retained as testing targets, and got real engineers at Volkswagen, Meituan, a Turkish startup, a Belgian logistics firm, and one anonymous company to write Overlays and run the tool. Table 1 is small but consistent in direction. The industrial quotes are useful and the limitations section is honest about short runs, fuzzer bugs on A3, and the fact that A4 was already fully covered. Citation pattern is normal for an EvoMaster-centric group; the Overlay claim itself is not circular.\n\nThe soft spot the stress-test flags is real but secondary. From the fuzzer’s point of view Overlay is just a clean way to inject the same examples you could paste into a schema copy. The experiment therefore measures “examples + Overlay delivery” against “no examples,” not Overlay against an alternative seed format. That does not kill the paper; the authors’ actual claim is viability and reduced vendor lock-in / cognitive load, not a pure causal isolation of the standard. Still, a third arm with identical examples injected another way would have made the contribution sharper. Single 10-minute runs with no variance are also thin, but the paper never pretends otherwise.\n\nWho it is for: people who build or deploy REST fuzzers in industry, and anyone who cares about standards-based tooling rather than yet another proprietary seed file. It is not a deep algorithmic advance, but it is reproducible engineering with multi-company evidence, which is rarer than it should be.\n\nI would send it to peer review. A referee can ask for the missing control arm or longer runs; the core idea and the industrial data are already worth the discussion.","headline":"Solid industrial feasibility study of Overlay as a portable way to feed examples into REST fuzzers; the Table 1 gains are real but do not isolate the standard from the hand-written examples.","tokens_in":12500,"tokens_out":518,"would_cite":true,"duration_ms":6056,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A standard Overlay format lets testers feed domain examples into REST API fuzzers without vendor lock-in, and industrial runs show it raises successful endpoint coverage.","keywords":["REST","API","fuzzing","black-box","OpenAPI","Overlay","testing","examples"],"falsifier":"Re-run the identical five APIs for many independent longer sessions (or with other fuzzers that only consume a post-merge schema) and show that the coverage and fault gains disappear once the same examples are supplied without Overlay.","tokens_in":12543,"feed_emoji":"🔧","tokens_out":808,"duration_ms":11778,"temperature":0.7,"pith_summary":"Black-box REST API fuzzers usually start from an OpenAPI schema alone, which is often too thin to produce the right parameter combinations or realistic identifiers. Testers therefore need a way to inject extra example values, yet custom per-fuzzer formats create lock-in and extra learning cost. This paper shows that the OpenAPI Initiative Overlay standard can carry those examples as named transformations on the schema. After native Overlay support was added to a mature fuzzer, engineers at five companies of widely different sizes wrote Overlay files for one of their own APIs and ran matched ten-minute sessions. In every case the Overlay-augmented runs matched or beat baseline coverage of successful responses and, on three APIs, also found additional faults. The work therefore argues that Overlay is already a practical, standards-based channel for domain knowledge in industrial black-box fuzzing.","feed_headline":"Overlay files lift REST API fuzzing on five live systems","feed_subtitle":"Standard transformations let testers inject domain examples without vendor lock-in, raising successful coverage.","key_machinery":"OAI Overlay actions that insert named “examples” (and same-name combinations) into an OpenAPI schema; the fuzzer samples those named values, treats each named combination-plus-status-code as an archive target, and surfaces their use in an interactive report.","core_discovery":"Industrial experiments on five live APIs demonstrate that supplying domain examples through OAI Overlay files measurably improves the black-box coverage and fault-finding power of a REST fuzzer relative to the same fuzzer run without those examples.","pith_inferences":["The same Overlay mechanism could later carry “links” or workflow sketches, turning it into a broader channel for domain knowledge beyond scalar examples.","Because Overlay is an independent standard, the same files could be shared across fuzzers, security scanners and mock servers once more tools add native support.","The modest gains on the smallest and on the buggy-fuzzer APIs suggest Overlay is most valuable precisely when the fuzzer’s own search is already competent but still data-starved."],"forward_implications":["Any fuzzer that already understands OpenAPI examples can immediately accept Overlay-supplied data without inventing a proprietary hint language.","Testers can keep example data in separate, version-controlled Overlay files that survive schema regenerations.","Named-example combinations give a lightweight way to express multi-parameter constraints that the schema itself does not capture.","Multiple Overlay files can be organised by feature area and applied together, scaling beyond a single monolithic file.","Generated test reports can highlight which supplied examples actually appeared, aiding review with product owners."],"fun_headline_variants":["OAI Overlay raises REST fuzzer coverage on five industrial APIs","Domain examples via Overlay improve black-box REST fuzzing results","Overlay files lift EvoMaster coverage and faults on five live APIs","Industrial runs show Overlay enhances black-box REST API testing","Testers inject examples through Overlay to boost REST fuzzer power"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That a single ten-minute run performed by the same engineers who wrote the Overlay files is enough to credit the Overlay mechanism itself, rather than the quality of the hand-chosen examples or the particular APIs chosen.","fun_headline_variants_meta":{"raw":{"variants":["OAI Overlay raises REST fuzzer coverage on five industrial APIs","Domain examples via Overlay improve black-box REST fuzzing results","Overlay files lift EvoMaster coverage and faults on five live APIs","Industrial runs show Overlay enhances black-box REST API testing","Testers inject examples through Overlay to boost REST fuzzer power"]},"model":"grok-4.5","effort":"low","cost_usd":0.00493,"raw_usage":{"total_tokens":1391,"prompt_tokens":758,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":49300000,"prompt_tokens_details":{"text_tokens":758,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":543,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":758,"tokens_out":90,"duration_ms":4433,"temperature":1.0,"reasoning_tokens":543,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T20:02:42.862336+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the identical five APIs for many independent longer sessions (or with other fuzzers that only consume a post-merge schema) and show that the coverage and fault gains disappear once the same examples are supplied without Overlay.","supporting_citations":[],"review_version":1}