{"id":"b9ec16c6-b5bb-4ba2-9eba-6652e6202092","arxiv_id":"2509.01612","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces standardized schemas for REST API authentication and fault reporting, a 36-API Docker-backed dataset, and a six-fuzzer comparison with guidelines for fair experimentation.","lead":"This paper presents a common format for authentication and fault reporting, plus a 36-API open-source benchmark, for REST API fuzzing research. It matters because fair, repeatable fuzzer comparisons are currently hard, and this is the largest open dataset proposed to fix that.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical comparison is confounded: only EvoMaster gets WFC auth config and crash fixes, so the study cannot demonstrate that WFD enables fair fuzzer comparisons.","rationale":"I agree with the reader's identification of the most load-bearing concern. The central claim of the paper is twofold: (1) WFC provides useful standards, and (2) WFD is a large dataset with scaffolding that supports fair comparisons. The first claim is well supported by the released schemas, libraries, and HTML report tooling, independent of the empirical study. The second claim is the one put to the test in Section 6, and that test is severely confounded. Only EvoMaster receives authentication credentials in the WFC format—the very format the paper introduces—and only EvoMaster receives crash fixes before the runs. These advantages are acknowledged in the text, but they directly align with the tool that the authors develop, making the large performance margin (average 57.2% endpoint coverage vs. 23.6% for RESTler and 17.8% for Schemathesis in Table 4) impossible to interpret as evidence of WFD's ability to produce fair comparisons. Additionally, EmRest runs on different hardware and misses two APIs, adding further confounds. The paper's own goal is not to declare a winner, but the empirical study is still presented as a demonstration of WFD's usefulness. A fairer comparison—or an explicit reframing of the study as a usability/robustness analysis rather than a comparative evaluation—would be needed to substantiate the fairness claim. Since the reader already issued a CONDITIONAL verdict with precisely this reasoning, I do not see a need to change the verdict. The concern is significant but not fatal to the artifact contributions, so accepting the conditional framing is appropriate.","tokens_in":49928,"tokens_out":2959,"duration_ms":37205,"concrete_test":"Rerun RESTler and Schemathesis on the 15 authentication-requiring APIs in WFD using their native authentication mechanisms (e.g., Python scripts as documented), while keeping EvoMaster's WFC-based configuration, all on the same hardware and with the same 10 repetitions. Then recompute the 2xx endpoint coverage and 500-fault metrics of Tables 4 and 5. If EvoMaster's average rank and margin over RESTler/Schemathesis shrink materially (or if EvoMaster no longer ranks first on a majority of these APIs), the reported comparison is largely a configuration artifact, and the fairness demonstration of WFD fails. A secondary check: rerun EmRest on the same Windows server (or all tools on macOS) to rule out hardware effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value proposition is that WFC/WFD standardize authentication and fault reporting and thereby support fair, large-scale fuzzer comparisons. Section 6 admits that authentication information was provided only in WFC format, currently supported only by EvoMaster, and that EvoMaster received crash fixes before the runs, while RESTler and Schemathesis were not given their native authentication scripts. EmRest additionally ran on different hardware and was missing 2 of 36 APIs. These are not minor details: the headline empirical result (EvoMaster far outperforming all others on 2xx coverage, 500-fault endpoints, and JaCoCo coverage in Tables 4–6) could be substantially driven by these asymmetries. The paper explicitly disclaims the goal of determining the 'best' fuzzer, but it still uses the comparison to draw conclusions about tool usability, schema robustness, and the value of WFD as a benchmarking infrastructure. If WFD cannot yield a fair comparison even in the authors' own showcase study, the claim that it 'facilitates fair comparisons among fuzzers' (Abstract) is undercut. The weakness is disclosed, but disclosure does not resolve the confound; it only narrows the claim to a usability demonstration. That narrower claim is still supported, but the broader fairness claim is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Web Fuzzing Commons (WFC), a set of JSON schemas and open-source libraries for declarative authentication configuration and standardized fault reporting for REST API fuzzers, and Web Fuzzing Dataset (WFD), a collection of 36 JVM-based open-source REST APIs with Docker scaffolding, authentication configs, and experiment scripts. To demonstrate usefulness, the authors run six fuzzers (ARAT-RL, EmRest, EvoMaster, LLamaRestTest, RESTler, Schemathesis) on all APIs for one hour each with 10 repetitions, measuring 2xx endpoint coverage, 500-status fault endpoints, and JaCoCo coverage. EvoMaster outperforms the others dramatically, but the authors acknowledge confounds: only EvoMaster received WFC authentication credentials and crash fixes, EmRest ran on different hardware and skipped two APIs. The paper discusses methodological pitfalls (cherry-picking APIs, measuring coverage during fuzzing versus on generated tests, lack of executable tests from many fuzzers, flaky assertions) and provides guidelines for future comparisons.","tokens_in":50229,"tokens_out":6923,"duration_ms":72150,"significance":"If adopted, WFC's authentication schema and fault catalog could reduce setup effort and improve comparability across fuzzers; WFD is the largest open-source REST API benchmark with scaffolding to date. The most valuable contribution may be the methodological discussion in Section 6.2 on coverage measurement and test generation, which is well illustrated with data. However, the empirical demonstration is weakened by confounds: the only fuzzer that supports WFC is the authors' own EvoMaster, and the comparison gives EvoMaster both authenticated access and targeted bug fixes. Thus the paper does not yet establish that WFD enables fair cross-tool comparisons, only that WFC works for EvoMaster. The guidelines are sensible, but they would be more convincing if followed in the paper's own showcase.","major_comments":[{"comment":"The comparison is not fair: WFC auth configs were provided only to EvoMaster, and EvoMaster crashes were fixed before the runs, while RESTler/Schemathesis did not receive their native auth scripts and other tools' crashes were not fixed. This contradicts the claim in §6.1 that 'all tools get exactly the same information as input.' The headline margins in Tables 4–6 (EvoMaster 57.2% avg 2xx vs. next 23.6%) are likely driven by these asymmetries. Since the paper presents WFD as facilitating fair comparisons (Abstract, §5.2), the authors must either supply equivalent auth to all tools (e.g., native auth for RESTler/Schemathesis) or explicitly restrict the claim to a demonstration of EvoMaster using WFC. This is load-bearing for the central value proposition.","section":"Section 6, 'Authentication information was provided...'"},{"comment":"EmRest was run on a different machine (MacBook Pro M4) than all other tools (Windows 11 server), and could not run genome-nexus and ocvn due to missing Docker images. Tables 4–6 still include zeros for these two APIs under EmRest, biasing the aggregated ranks and effect sizes. Missing data should be excluded or reported separately. This is a second confound in the headline comparison.","section":"Section 6, EmRest hardware and missing APIs"},{"comment":"The usefulness demonstration is circular: WFD is an extension of the authors' EMB, and the study fixes bugs only in EvoMaster. The paper acknowledges this bias, but the consequence is that the claim 'any fuzzer can benefit from WFC and WFD' (Abstract) is unsupported. To substantiate this, the authors should integrate WFC into at least one independent fuzzer (or provide a detailed integration guide with a case study). Without this, the evaluation only shows that EvoMaster can use WFC/WFD, not that the artifacts are generally usable.","section":"Section 6.1, 'As the author of EvoMaster...'"}],"minor_comments":[{"comment":"The pairwise comparison labels X and Y are not identified in the text; although the paper says 'which tools they are is not relevant', readers are left guessing (EvoMaster is X and Schemathesis is Y). Please name them for transparency.","section":"Section 6.2, Table 7"},{"comment":"The 'Total' row reports 6,465 source files and 657,162 LOCs, but summing the visible rows appears inconsistent. Please verify the totals.","section":"Section 5.2, Table 3"},{"comment":"Typo: 'Representational Sate Transfer' should be 'Representational State Transfer'.","section":"Section 2.1"},{"comment":"The fault catalog includes F100/F101 etc., but only EvoMaster's results are shown. It would be helpful to state explicitly that the other fuzzers do not yet produce WFC-format reports, so the table does not permit cross-tool fault comparisons.","section":"Section 6.1, Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper is authored by the EvoMaster team, and the self-evaluation is a concern for a journal that values independent validation. The artifacts are valuable, but the empirical section needs strengthening before publication. Given the paper's length, the guidelines and dataset could potentially be separated into a tool/benchmark paper with a more circumscribed evaluation. The referees should weigh the disclosed confounds carefully when assessing the soundness of the empirical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the thing to know: WFC/WFD are real deliverables. The schemas, the Docker and authentication scaffolding, and the 36-API corpus are open, documented, and would plausibly lower the cost of running and comparing REST API fuzzers. Engage with the paper on those grounds, not because of its empirical ranking.\n\nWhat is new: a declarative auth schema, a fault catalog with reserved code ranges, a report JSON schema with an HTML viewer, and the dataset itself. The guidelines about tool usability and cherry-picking are sensible, and the demonstration that cherry-picking a few APIs can flip the winner is a nice object lesson. The authors are transparent about where their own tool gets special treatment.\n\nWhere the soft spots are: the six-tool comparison cannot bear the weight the paper tries to put on it. Only EvoMaster received WFC auth credentials and had crashes fixed before the final runs; EmRest ran on a different OS and two APIs could not run under it; RESTler and Schemathesis were not given their native auth scripts. The paper discloses all of this, and even says the goal is not to pick a best tool, but the abstract still claims the dataset “facilitates fair comparisons among fuzzers.” As a demonstration of WFD’s usefulness, fine. As evidence of relative tool performance, it is not usable. That should be reframed or fixed with a follow-up where each fuzzer gets its native auth setup and a common platform.\n\nMinor: the fault catalog is useful but the counts for non-EvoMaster tools are mostly zero, so fault-based comparison is even more lopsided than the coverage numbers. The threats-to-validity section covers randomness and generalization but does not resolve the asymmetry.\n\nWho this is for: anyone building or evaluating REST API fuzzers, and any reviewer of empirical software engineering work. It deserves a serious referee. I would send it out, but tell the authors the empirical section needs either native-auth re-runs or an explicit removal of comparative claims. The dataset alone is worth citing.","headline":"Useful standard-and-dataset paper whose engineering artifacts matter more than its own head-to-head fuzzer results, which are too confounded to interpret.","tokens_in":50716,"tokens_out":1714,"would_cite":true,"duration_ms":21328,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes WFC (schema libraries for authentication and fault reporting) and WFD (36 authenticated REST APIs with Docker scaffolding), arguing these remove three chronic obstacles to fair REST API fuzzing experiments: authentication","keywords":["REST API fuzzing","authentication specification","fuzzer benchmark","fault catalog","empirical comparison","OpenAPI","reproducibility","test generation"],"falsifier":"Re-run the 15 auth-requiring APIs in WFD after giving RESTler and Schemathesis their native script-based authentication setups (and keeping the other tools as-is). If the coverage and fault-detection gap between EvoMaster and those two tools collapses or reverses, the paper's implicit claim that WFD exposes intrinsic tool weaknesses would not hold. A second check: feed each non-EvoMaster fuzzer pre-fixed OpenAPI schemas that resolve basePath/server URL ambiguities, to see if the 'schema robustness' explanation for their near-zero results is correct.","tokens_in":49802,"feed_emoji":"🔑","tokens_out":3980,"duration_ms":46264,"temperature":0.7,"pith_summary":"REST API fuzzing research keeps tripping on three practical problems: how a fuzzer learns to log in, how different tools report which faults they found, and which APIs everyone should agree to test on. This paper tries to turn each problem into a standardized, reusable artifact. WFC is an open-source library of JSON Schema definitions: one schema declaratively describes authentication information (static credentials or dynamic token extraction from login endpoints), and another defines a fuzz-report format with a catalog of fault types. WFD is a corpus of 36 open-source JVM REST APIs, each with Docker Compose setup, coverage tooling, and WFC-formatted authentication configs, making it the largest open-source REST API selection used in academic fuzzing experiments to date.","feed_headline":"One shared spec for API logins could make fuzzer trials fair","feed_subtitle":"WFC/WFD give fuzzers one way to declare authentication and report faults, so tool comparisons stop being apples-to-oranges.","key_machinery":"The central objects are two schema definitions packaged as WFC: auth.yaml, a declarative schema for authentication configuration (static credentials, login endpoint payloads, token extraction via JSON Pointer, cookie or header injection), and report.yaml, a schema for fuzzer output with a catalog of fault-type codes for oracles found in the literature. The second machinery piece is WFD, a corpus of 36 JVM REST APIs with Docker Compose scaffolding, JaCoCo instrumentation, mitmproxy HTTP logging, and WFC auth files for the 15 APIs that require login. Together they let a fuzzer be pointed at an API with the same declarative auth config and produce reports that other tools and HTML viewers can r","core_discovery":"The authors claim that the three major barriers to rigorous REST API fuzzing comparisons are plumbing problems rather than research gaps: there is no standard way to tell a fuzzer how to authenticate, no standard taxonomy or format for reporting detected fault types, and no shared, scaffolded corpus of real APIs for experiments. WFC attacks the first two by publishing JSON Schemas: an auth schema that declaratively describes static credentials and dynamic token endpoints (including cookie vs. Authorization-header extraction), and a report schema that assigns unique IDs to automated oracles (e.g., F100 for HTTP 500, F101 for schema-validation mismatches, F204–F206 for access-policy violations","pith_inferences":["If RESTler and Schemathesis were given WFC-equivalent credentials in their native script-based form on the 15 auth-requiring WFD APIs, the reported EvoMaster margin could shrink considerably; that is a directly testable re-run.","The roughly 20% assertion-failure rate in EvoMaster's generated Python tests points toward flaky-assertion and test-order research, and suggests mutation testing could become a standard quality gate for REST API fuzzers.","The community-superset selection rule the paper advocates for REST APIs could transfer to other software testing domains, making 'add new subjects rather than drop old ones' a general expectation for empirical evaluations.","WFD's JVM-only composition leaves open whether the findings generalize to non-Java REST stacks; extending the corpus with Python, Go, or Node.js APIs would test that scope."],"forward_implications":["If fuzzers adopt WFC, authentication configuration written once for an API can be reused across tools, removing a major source of setup effort and vendor lock-in.","WFD gives future studies a common 36-API corpus with working scaffolding, making cherry-picked API subsets harder to justify and enabling community-driven superset growth.","A standardized fault report format allows comparisons that go beyond HTTP 500 counts, capturing security-relevant faults such as access-policy violations.","The comparison guidelines imply that fuzzer evaluations should measure coverage on the final generated executable tests, not just during the fuzzing session, because the session-time numbers overstate what practitioners receive.","The reserved fault-code range (900–999) lets new experimental oracles be reported in WFC format before they are formally added to the catalog."],"supporting_citations":[{"why":"Source of the declarative authentication approach and the fuzzer used as the WFC/WFD proof-of-concept.","marker":"[20]"},{"why":"The EMB corpus that WFD extends; supplies the prior API collection and scaffolding approach.","marker":"[18]"},{"why":"Industrial REST API fuzzing experience motivating declarative authentication and robustness requirements.","marker":"[15]"},{"why":"Independent 10-fuzzer comparison on 20 APIs; basis for selecting EvoMaster and for including those APIs in WFD.","marker":"[57]"},{"why":"Authors' earlier fuzzer comparison on EMB, which this paper extends and critiques methodologically.","marker":"[86]"},{"why":"RESTler, a popular fuzzer compared in the study; its script-based authentication is the contrast WFC replaces.","marker":"[23]"},{"why":"Schemathesis, another compared fuzzer; also uses script-based authentication, forming the comparison baseline.","marker":"[49]"},{"why":"Source of access-policy-violation oracles that become fault categories F204–F206 in the WFC catalog.","marker":"[16]"}],"fun_headline_variants":["Standardizing auth and fault reporting to end unfair API fuzzing tests","Shared specs for auth and faults could make API fuzzing comparisons fair","WFC/WFD: One library to rule REST API fuzzing experiments","How to make REST API fuzzer head-to-heads actually comparable","Open-source commons and dataset to standardize REST API fuzzing"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The empirical comparison assumes the large performance gap reflects the fuzzers' intrinsic capabilities, even though only EvoMaster received authentication credentials in WFC format and received crash fixes before the runs, while the other tools ran without their native authentication configurations.","fun_headline_variants_meta":{"raw":{"variants":["Standardizing auth and fault reporting to end unfair API fuzzing tests","Shared specs for auth and faults could make API fuzzing comparisons fair","WFC/WFD: One library to rule REST API fuzzing experiments","How to make REST API fuzzer head-to-heads actually comparable","Open-source commons and dataset to standardize REST API fuzzing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3410,"prompt_tokens":800,"completion_tokens":2610,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2514}},"tokens_in":544,"tokens_out":2610,"duration_ms":17931,"temperature":1.0,"reasoning_tokens":2514,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:21:11.078376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 15 auth-requiring APIs in WFD after giving RESTler and Schemathesis their native script-based authentication setups (and keeping the other tools as-is). If the coverage and fault-detection gap between EvoMaster and those two tools collapses or reverses, the paper's implicit claim that WFD exposes intrinsic tool weaknesses would not hold. A second check: feed each non-EvoMaster fuzzer pre-fixed OpenAPI schemas that resolve basePath/server URL ambiguities, to see if the 'schema robustness' explanation for their near-zero results is correct.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the declarative authentication approach and the fuzzer used as the WFC/WFD proof-of-concept."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The EMB corpus that WFD extends; supplies the prior API collection and scaffolding approach."},{"cited_title":"Arcuri, A","cited_arxiv_id":null,"evidence_quote":"Industrial REST API fuzzing experience motivating declarative authentication and robustness requirements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Independent 10-fuzzer comparison on 20 APIs; basis for selecting EvoMaster and for including those APIs in WFD."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RESTler, a popular fuzzer compared in the study; its script-based authentication is the contrast WFC replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Schemathesis, another compared fuzzer; also uses script-based authentication, forming the comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of access-policy-violation oracles that become fault categories F204–F206 in the WFC catalog."}],"review_version":1}