{"id":"fc48820a-bda6-4453-8d17-9440a306bdd5","arxiv_id":"2507.20848","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A white-box fuzzer for REST APIs was extended with MongoDB-aware heuristics and direct database insertion, improving code coverage on six APIs by up to 18% over the base tool.","lead":"This paper extends the EvoMaster fuzzing tool to understand and manipulate MongoDB databases during automated testing of REST APIs. The tool reads the database's state to guide test generation and can insert data directly, which improved code coverage on six Java APIs by up to 18% compared to the base tool.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the novel NoSQL distance heuristic from direct database insertion, so the reported RQ1 gains cannot be attributed to the paper's primary technical contribution.","rationale":"The paper is a solid empirical contribution with a clear tool, a reproducible setup, and unusually strong effect sizes. My concern is not that the combined approach fails, but that the central causal claim is underdetermined. The reader's weakest_assumption points to the heuristic's uncertain influence on coverage; I sharpen this into an attribution problem: because direct insertion and the heuristic are never separated, the RQ1 results cannot validate the heuristic specifically. This is concrete and testable, and it aligns with the reader's call for an ablation study. It does not overturn the conditional verdict; it reinforces the condition under which the paper's novelty claims should be accepted. I therefore recommend no change to the verdict.","tokens_in":19580,"tokens_out":4438,"duration_ms":55498,"concrete_test":"Run a 2x2 ablation on the same six EMB APIs with 30 one-hour sessions per configuration: (A) Base, (B) Mongo with NoSQL distance heuristic disabled but direct insertion enabled, (C) Mongo with heuristic enabled but direct insertion disabled, and (D) full Mongo. Compare line coverage and detected faults using the same EvoMaster reporting. If B and D are statistically indistinguishable while A is worse than both, the novel heuristic contributes nothing beyond direct insertion; if C is close to A, direct insertion alone explains the headline improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RQ1 comparison contrasts Base with Mongo, where Mongo enables two mechanisms at once: the NoSQL distance heuristic (Section 5) and direct MongoDB insertion (Section 6). Direct insertion can trivially satisfy find() filters by writing documents before the SUT is called, and this mechanism alone is known to help in the SQL setting from which it is adapted [22]. The distance heuristic, by contrast, is only a secondary objective and the paper itself concedes in Section 5.3 that having data returned by the database is not guaranteed to improve coverage. Since both mechanisms are always enabled in the Mongo configuration, the observed A12 = 1.00 on four of six APIs does not establish that the MongoDB-specific heuristic contributes anything. Without an ablation separating the two, the empirical support for the paper's first stated novelty is missing, even though the combined tool may still be effective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an extension of EvoMaster for white-box fuzzing of REST APIs that interact with MongoDB. Two mechanisms are introduced: (i) runtime instrumentation that monitors MongoCollection.find() calls and computes a \"NoSQL distance\" heuristic approximating how close a collection's existing documents are to satisfying a filter that returned no results, used as a secondary objective (§5.1–§5.3); and (ii) direct generation of MongoDB insertion calls from test cases, with a schema inferred reflectively from the expected result classes observed at find() sites (§6). The approach is evaluated on six JVM-based REST APIs from the EMB corpus that use MongoDB. RQ1 compares the extended tool (Mongo) against default EvoMaster (Base) over 30 one-hour runs per API, reporting statistically significant coverage gains on five of six APIs with average A12=0.91 and up to 18% average improvement, plus increased fault detection on three APIs. RQ2 compares against four black-box fuzzers (ARAT-RL, EvoMaster BB, Restler, Schemathesis), where Mongo obtains the best average rank (1.5) and best average JaCoCo line coverage (52.7%). The paper includes a replication package.","tokens_in":19743,"tokens_out":6813,"duration_ms":69709,"significance":"The work addresses a real and increasingly relevant problem: REST APIs whose control flow depends on data stored in schema-less NoSQL databases, where standard branch-distance heuristics give no gradient. The proposed instrumentation at the MongoDB driver level is a reasonable, ODM-independent design. The empirical study is solid in several respects: 30 repetitions, one-hour sessions, standard statistical tests, and honest reporting of cases where the method did not win (e.g., ocvn-rest, and session-service vs ARAT-RL). The authors also disclose the secondary nature of the NoSQL distance objective and the fact that a higher cost per test evaluation leads to fewer HTTP calls. The open-source implementation and replication package are valuable strengths. The main weakness is that the experimental design conflates the two proposed mechanisms, so the evidence for the NoSQL distance heuristic—explicitly listed as a first novelty in the introduction—is missing. This is fixable with an ablation, but until that is done the paper's strongest claims should be read as being about the combined technique.","major_comments":[{"comment":"The Base-vs-Mongo comparison does not isolate the NoSQL distance heuristic (§5) from direct MongoDB insertion (§6). In the Mongo configuration both mechanisms are always enabled, and direct insertion alone can trivially make a previously empty find() return data by writing documents before the SUT is called; this mechanism is an adaptation of the SQL technique in [22], which the paper itself cites as effective. Because the paper's first stated contribution is the NoSQL heuristic, and because §5.3 concedes that \"we cannot be sure that having data returned by the database will necessarily have a beneficial impact on achieving coverage of g,\" the observed A12=1.00 on four of six APIs cannot be attributed to that heuristic. An ablation (e.g., a configuration with the heuristic but without direct insertion, and/or direct insertion without the heuristic) is needed to support the contribution as stated; if the authors intend to claim only the combined technique, the claims and title should be adjusted accordingly.","section":"§7.3, Table 5, RQ1"},{"comment":"The paper reports the average number of detected faults for Base and Mongo but never describes what constitutes a detected fault or which oracles were used. EvoMaster has several built-in fault oracles, and the choice of oracle materially affects the counts. Without this information the RQ1 statement \"more faults are found\" cannot be interpreted or reproduced. Please specify the oracle configuration, the counting procedure, and whether faults were counted per unique failing HTTP call or per root cause.","section":"§7.3, Table 5, RQ1"}],"minor_comments":[{"comment":"The coverage metric differs between the RQ1 and RQ2 experiments (EvoMaster's own reporting tooling for RQ1 vs JaCoCo on generated test suites for RQ2). Please state explicitly which metric underlies the percentages in the abstract and in each table, and discuss the implications for comparing effect sizes across the two research questions.","section":"§7.2–§7.3"},{"comment":"The definition of Hd('f', C) relies on branch distance rho for string equality, but Table 1 only defines rho for numeric operands; a brief sentence describing how Levenshtein distance is combined with the numeric branch distance would remove ambiguity.","section":"§5.2, Table 3"},{"comment":"The heuristic is only computed when the target collection is non-empty; when the collection is empty, no gradient is available. This design choice, combined with the probabilistic insertion described in §6, further suggests that the direct-insertion mechanism may be the dominant driver of the reported results and should be examined in the ablation.","section":"§5.2 and §6"},{"comment":"The Friedman test is reported globally, but no post-hoc pairwise comparisons are given; the claim that Mongo is the best tool rests on average ranks and coverage averages, which would benefit from pairwise effect sizes.","section":"§7.3, Table 6"},{"comment":"There are several typos and nonstandard spellings: \"dependant\" should be \"dependent,\" \"Surpringly\" should be \"Surprisingly,\" and \"standarized\" should be \"standardized.\"","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper shares authorship with the EvoMaster/EMB maintainers and uses EMB as the SUT source; this is normal in tool papers, but the novelty of the MongoDB heuristic relative to the SQL adaptation in [22] needs more scrutiny than the prose gives it. The missing ablation is the decisive issue: if the authors can run even a partial ablation (e.g., a subset of APIs with 30 repetitions each), the paper would be substantially stronger. The fault-detection reporting also needs more detail before the RQ1 fault claims can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a solid piece of empirical SE work. It extends EvoMaster, the only white-box REST API fuzzer, with MongoDB support: driver-level monitoring of find() calls, NoSQL distance heuristics for empty-result queries, and direct insertion of BSON documents with schema inference. That last part is genuinely new in the NoSQL setting, since MongoDB lacks a fixed schema. The engineering is real: open-source extension, replication package, and 45 days of experiments on six EMB APIs with 30 repetitions. The stats follow the field's conventions, and the authors report cases where they lose (ARAT-RL beats them on two APIs), which they did not have to do.\n\nThe main result is believable. RQ1: average A12=0.91, four of six APIs at 1.00, up to 18% coverage gain. RQ2 is less important but shows the combined tool is competitive with state-of-the-art black-box fuzzers.\n\nWhere the soft spot is: the RQ1 contrast is Base vs Mongo, and Mongo turns on two mechanisms at once — the new NoSQL distance heuristic (Section 5) and direct data insertion (Section 6). There is no ablation. As the stress-test note says, direct insertion alone can trivially satisfy find() filters by writing documents before the SUT is called, and the distance heuristic is only a secondary objective. The paper even concedes in Section 5.3 that having data returned is not guaranteed to help coverage. So the paper does not establish that the MongoDB-specific distance heuristic contributes anything. The combined tool works, but the first stated novelty — the NoSQL heuristics — is not isolated. This is a real weakness, in proportion: it reduces the contribution's attribution rather than negating the empirical result. The authors should add an ablation (Base+insertion vs Base+insertion+heuristic) before acceptance.\n\nMinor points: only six APIs, all JVM, one of them write-heavy which dilutes the effect, and the black-box comparison coverage measurement differs between tools (JaCoCo during fuzzing vs on generated JUnit tests). The paper acknowledges these.\n\nOverall: this paper deserves a serious referee. The work is reproducible and honest, and the combined approach likely helps practitioners. The citation pattern looks fine; the prior SQL work is cited and the differences are real. My own verdict is conditional on the ablation. For peer review, I'd accept and push for an ablation experiment.","headline":"Solid empirical extension of EvoMaster to MongoDB; just don't take the RQ1 gains as evidence for the distance heuristic until it's isolated from direct insertion.","tokens_in":20252,"tokens_out":2008,"would_cite":true,"duration_ms":21721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MongoDB-aware fuzzing significantly improves white-box REST API testing, with max effect size on four of six APIs.","keywords":["REST API fuzzing","search-based software testing","MongoDB","NoSQL database","test case generation","code coverage","white-box testing","database heuristics"],"falsifier":"Run a controlled ablation on the same six APIs under the paper's one-hour, 30-repetition protocol, comparing Base, Base plus the NoSQL distance only, Base plus direct document insertion only, and the full combination. If the distance-only variant shows no coverage gain over Base while the full variant keeps $\\hat{A}_{12}=1.00$ on the same four APIs, the heuristic component is not the load-bearing mechanism and the paper's explanation of the gain would need revision.","tokens_in":1722,"feed_emoji":"🗄️","tokens_out":3061,"duration_ms":123102,"temperature":0.7,"pith_summary":"The paper's thesis is that a white-box fuzzer for REST APIs can be substantially more effective for MongoDB-backed services if it stops treating the database as a black box. It proposes two concrete mechanisms: a NoSQL distance that scores how close each empty query result is to returning documents, and a way for generated test cases to insert documents straight into MongoDB with field types inferred from the API's own result classes. The authors argue this matters because the database boundary blocks ordinary branch-distance guidance, and because many microservices cannot create via HTTP the data they serve, making direct insertion the only practical route. On six real-world APIs, the extended fuzzer improves average line coverage from 40.0% to 47.3%, with the maximum effect size (every Mongo run better than every baseline run) on four of the six and a peak gain of 18%.","feed_headline":"MongoDB-aware fuzzing lifts REST API coverage 18%","feed_subtitle":"NoSQL-aware guidance and direct DB writes outperform the base fuzzer on five of six real APIs.","key_machinery":"The load-bearing device is a NoSQL distance $H_c(F)=\\min_{d\\in c}H_d(F)$: for each filter $F$ observed on a find() call that returned no documents, the instrumented tool re-executes the find and scores every stored document by recursively translating the filter's operators into branch-distance-style quantities (using the standard branch distance $\\rho$ and normalizing conjunctions with $\\nu(x)=x/(x+1)$), so that a score of zero means the document satisfies the filter. This distance is added as a secondary objective in the search, breaking ties among test cases with equal primary coverage. A second device is direct document insertion: the fuzzer emits a MongoDB insertion call whose JSON payload is generated from field types inferred reflectively from the repository's result class, so the inserted document matches the format the API expects to read.","core_discovery":"The discovery is that MongoDB interactions can be turned from an opaque sink for search gradient into a measurable secondary objective. The tool instruments every call to the MongoDB driver's find() method, records filters that return no documents, and after each test execution re-runs those filters to compute a distance $H_c(F)=\\min_{d\\in c}H_d(F)$ over the documents currently in the collection; this distance is added to the fuzzer's fitness function as a tie-breaker. Alongside this, a new initialization call lets a generated test insert a JSON document into a specified collection, with the document's fields sampled to match the types inferred by reflection from the repository's result class, so the inserted data satisfies the implicit schema. The empirical claim is that, compared with the default white-box tool, this combination raises line coverage on five of the six studied APIs (with average effect size $\\hat{A}_{12}=0.91$), finds more faults on three, and does so while generating tests that issue fewer HTTP calls on average.","pith_inferences":["Editorial inference: an ablation separating the two mechanisms would tell practitioners whether direct insertion alone, without the distance heuristic, already explains the $\\hat{A}_{12}=1.00$ results; if so, that would be the cheaper configuration to adopt.","Editorial inference: because the distance is computed after the test run on a database state that may have changed, a real-time or incremental version of the heuristic could provide a more stable gradient for write-heavy APIs.","Editorial inference: the same filter-distance idea could be turned into a test oracle, searching for documents that satisfy a query's negation and thereby trigger error-handling paths, rather than only using the distance as a coverage guide."],"forward_implications":["On five of the six studied APIs, the MongoDB-aware variant beats the default white-box fuzzer in line coverage, with average effect size $\\hat{A}_{12}=0.91$ and the maximum $\\hat{A}_{12}=1.00$ on four APIs, meaning every one of the 30 runs was better than every baseline run there.","Average line coverage rises from 40.0% to 47.3% across the six APIs, with the largest gain from 57.3% to 75.7% on one API.","The technique detects more faults on three of the six APIs, with an average effect size of $\\hat{A}_{12}=0.70$ for fault count.","Because monitoring happens at the MongoDB Java driver level, the approach works regardless of the object-document mapper the API uses internally.","Generated tests need fewer HTTP calls on average to reach higher coverage, since data can be inserted directly instead of being built through lengthy endpoint sequences."],"supporting_citations":[{"why":"Supplies the SQL-database method this work extends: secondary objectives for empty queries and direct database insertion from test cases.","marker":"[22]"},{"why":"Gives the bytecode-instrumentation approach for monitoring calls to MongoCollection.find() at the Java driver level.","marker":"[25]"},{"why":"Provides the curated corpus of APIs from which the six studied JVM-based MongoDB systems were selected.","marker":"[26]"},{"why":"Defines the Wilcoxon-Mann-Whitney and A12 effect-size analysis protocol used to compare the 30-run experiments.","marker":"[21]"},{"why":"Describes the white-box search engine and branch-distance heuristics that the new techniques extend.","marker":"[19]"},{"why":"Introduces the search-based test generation tool in which the MongoDB extensions are implemented.","marker":"[17]"},{"why":"Supplies the normalization function $\\nu(x)=x/(x+1)$ used inside the NoSQL distance to combine sub-scores.","marker":"[15]"}],"fun_headline_variants":["MongoDB-aware fuzzing lifts REST API coverage 18%","NoSQL state tracking improves REST API fuzzing coverage","EvoMaster extension uses MongoDB state to boost fuzzing","White-box fuzzing with MongoDB awareness finds more faults","Direct DB writes and state analysis enhance REST fuzzing"],"cache_read_input_tokens":22528,"weakest_assumption_plain":"The load-bearing premise is that rewarding test cases whose empty MongoDB queries come closer to returning data will, often enough, translate into higher code coverage and more fault detection; the paper itself flags this as an uncertain secondary objective.","fun_headline_variants_meta":{"raw":{"variants":["MongoDB-aware fuzzing lifts REST API coverage 18%","NoSQL state tracking improves REST API fuzzing coverage","EvoMaster extension uses MongoDB state to boost fuzzing","White-box fuzzing with MongoDB awareness finds more faults","Direct DB writes and state analysis enhance REST fuzzing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2644,"prompt_tokens":967,"completion_tokens":1677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1592}},"tokens_in":583,"tokens_out":1677,"duration_ms":14187,"temperature":1.0,"reasoning_tokens":1592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:10:28.997285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled ablation on the same six APIs under the paper's one-hour, 30-repetition protocol, comparing Base, Base plus the NoSQL distance only, Base plus direct document insertion only, and the full combination. If the distance-only variant shows no coverage gain over Base while the full variant keeps $\\hat{A}_{12}=1.00$ on the same four APIs, the heuristic component is not the load-bearing mechanism and the paper's explanation of the gain would need revision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SQL-database method this work extends: secondary objectives for empty queries and direct database insertion from test cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the curated corpus of APIs from which the six studied JVM-based MongoDB systems were selected."},{"cited_title":"Arcuri and L","cited_arxiv_id":null,"evidence_quote":"Defines the Wilcoxon-Mann-Whitney and A12 effect-size analysis protocol used to compare the 30-run experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the white-box search engine and branch-distance heuristics that the new techniques extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the search-based test generation tool in which the MongoDB extensions are implemented."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the normalization function $\\nu(x)=x/(x+1)$ used inside the NoSQL distance to combine sub-scores."}],"review_version":1}