{"id":"e3c9f9b0-5e0c-46f3-8f50-c2dbe9281485","arxiv_id":"2501.08600","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AutoRestTest combines semantic dependency graphs, multi-agent reinforcement learning, and LLM-generated inputs to test REST APIs, and reports higher successful operation coverage than four baseline tools in preliminary runs.","lead":"AutoRestTest is a new tool that combines a dependency graph, five reinforcement-learning agents, and a large language model to automatically test REST APIs. If its preliminary results hold, it could find more working API operations and real server errors than existing test generators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OhSome result, which drives the claimed advantage, may be an artifact of unconfigured authentication in the baselines rather than evidence for the proposed architecture.","rationale":"I read the tool paper in good faith. The architecture is coherent: SPDG, five Q-learning agents, LLM value generation, and a request generator with a mutator. The repository, CLI, report outputs, and demonstration video are concrete artifacts that support replication. The central claim, however, is an empirical outperformance claim, and that claim is only as strong as the comparison. The comparison is presented in a single table with no variance, no run counts, and no baseline configuration details. More importantly, the largest part of the claimed advantage is concentrated in OhSome, where all four baselines produced zero 2xx responses. Because the paper itself identifies a header agent that supplies authentication tokens and calls it a recent addition, the natural hypothesis that OhSome requires authentication and that the baselines were not configured for it must be eliminated before the outperformance can be credited to the proposed approach. This does not require rejecting the paper; it requires a concrete replication step. The reader's conditional verdict is appropriate, and the condition should explicitly include authentication-equivalence for the baselines. My agreement with the reader is partial: I share the concern about baseline configuration, and I make it more specific by identifying the header-agent/authentication mechanism.","tokens_in":6149,"tokens_out":4649,"duration_ms":48516,"concrete_test":"On the same OhSome instance used in Section IV, configure RESTler, EvoMaster, ARAT-RL, and MoRest with the same API credentials and any required headers that AutoRestTest's header agent would generate, or with the authentication setup provided by the benchmark infrastructure from reference [8], and rerun each within the same one-hour budget, ideally 5-10 times. If any baseline obtains non-zero 2xx coverage, the claim that AutoRestTest is the only tool able to handle OhSome is not supported as stated; if all still obtain zero 2xx responses, the authentication confound is ruled out.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in Section IV's Table I: OhSome accounts for 12 of AutoRestTest's 26 successful operations, while all four baselines score 0 on that service. The paper never states whether OhSome's endpoints require authentication. If they do, the comparison is structurally unfair: Section II-B introduces a Header Agent that 'leverages account-related operations from the specification to supply basic token authentication headers,' and a footnote notes this agent is a recent addition not described in the companion research paper. The paper does not report whether RESTler, EvoMaster, ARAT-RL, or MoRest were supplied with equivalent credentials or header-generation facilities. Without that information, the headline 26-vs-12/11/11/10 difference could be caused by an unconfigured auth requirement rather than by the SPDG+MARL+LLM architecture that the paper claims to validate. The absence of run counts and variance data compounds this, but the auth confound is the sharper, testable risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoRestTest, a tool for automated REST API testing that combines a Semantic Property Dependency Graph (SPDG), five specialized agents (operation, parameter, value, dependency, header), Q-learning, and LLM-based value and header generation. The authors evaluate AutoRestTest against RESTler, EvoMaster, ARAT-RL, and MoRest on four real-world services (FDIC, OMDb, OhSome, Spotify) and report that AutoRestTest successfully processed 26 unique operations versus 12, 11, 11, and 10 for the baselines, and that it was the only tool to obtain 2xx responses on OhSome and the only one to detect a 5xx error on Spotify. The paper also describes the tool's configuration options, CLI, and report generation. The reported results are described as preliminary.","tokens_in":6370,"tokens_out":3742,"duration_ms":39948,"significance":"If the reported results are reproducible and the differences are not due to configuration or stochastic variation, AutoRestTest would be a useful contribution to REST API testing, particularly in combining dependency modeling, multi-agent RL, and LLM-driven input generation. The artifact is publicly available, the tool addresses a practically relevant problem, and the evaluation uses four existing baseline tools and real-world services. However, the empirical evidence as presented is not yet sufficient to support the central claim of superiority: the comparison lacks run counts and variance, baseline configuration details are absent, and the OhSome result introduces a plausible authentication confound that is not addressed.","major_comments":[{"comment":"The headline superiority claim relies heavily on the OhSome service, where AutoRestTest reports 12 successful operations and all four baselines report 0. The manuscript does not state whether OhSome's endpoints require authentication, nor whether RESTler, EvoMaster, ARAT-RL, and MoRest were provided with equivalent credentials or header-generation facilities. Since Section II-B describes a Header Agent that supplies token authentication headers and a footnote states that this agent is a recent addition not described in the companion paper, the comparison is confounded unless authentication handling is demonstrated to be equal across tools. Please report the authentication requirements of each service and the exact authentication-related configuration supplied to each baseline.","section":"Section IV, Table I"},{"comment":"All effectiveness values appear to come from a single run per tool and service. No run counts, variance, confidence intervals, or statistical tests are reported. Given the use of epsilon-greedy Q-learning, random request mutation, and LLM sampling at temperature 0.7, there is substantial stochasticity in AutoRestTest; a lucky run could plausibly explain the observed differences. The paper should report at least several runs per tool and service with means and spread, and should state whether the one-hour window is wall-clock time and how LLM latency is accounted for.","section":"Section IV"},{"comment":"The components of AutoRestTest are not validated independently. The SPDG construction depends on a similarity threshold of 0.7 and a top-three fallback; the Q-learning parameters are set to 0.1 and 0.9 based on prior work; and the LLM engine is not specified for the experiments. No ablation is provided to isolate the contribution of the SPDG, the MARL agents, the LLM, or the header agent. As a result, even if the comparison against the baselines were sound, the paper would not support the claim that the proposed architecture as a whole is responsible for the observed effectiveness. Please add an ablation or at least controlled variants that disable individual components.","section":"Sections II-A and II-C"},{"comment":"The header agent is described only as leveraging 'account-related operations from the specification to supply basic token authentication headers,' and the footnote explicitly states that this agent is not described in the companion research paper. Because the header agent may be responsible for the OhSome successes and for the only detected Spotify 5xx error, its behavior is load-bearing for the evaluation. The paper should specify how account-related operations are identified, how tokens are obtained, and how the resulting headers are attached to requests, or it should point to a precise code location so that the evaluation is reproducible.","section":"Section II-B, Footnote 1"}],"minor_comments":[{"comment":"The abstract and introduction describe 'five specialized agents,' but Section II-B says the REST agents 'consist of four specialized components' and does not count the dependency agent in that list. Please clarify the agent count and the placement of the dependency agent in the architecture.","section":"Abstract and Section II-B"},{"comment":"The configuration section describes LLM temperature and engine selection but the experimental evaluation in Section IV does not state which LLM engine, model version, and temperature were used. This information is needed for reproducibility.","section":"Section III-A-2"},{"comment":"The example report in Listing 1 appears to contain keys with extra spacing, such as 'T i t l e' and 'D u r a t i o n'; if the actual tool output is valid JSON, please ensure the listing reproduces it faithfully.","section":"Listing 1"},{"comment":"Reference [6] is cited in the introduction as an example of reinforcement learning for REST API testing, but the text does not state its publication venue or year; please complete the bibliographic details in the citation if possible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a tool-presentation paper whose central contribution is an empirical effectiveness claim. The missing baseline configuration details and run-level data are the main blockers; the OhSome authentication question is the sharpest testable risk. I would ask the authors to supply the missing data and clarify the authentication setup before considering the paper further. The paper is probably best suited to a tool-demo track, but the current claims go beyond a demonstration and need stronger evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, AutoRestTest is a real implemented tool — the paper ships a public repo and a detailed report format, and the architecture (SPDG, five reinforcement-learning agents, LLM-driven value/header generation) is coherent. Second, the headline empirical result ('26 unique operations vs 12/11/11/10') is not yet trustworthy as stated. The evaluation in Section IV is a one-hour, single-shot comparison with no run counts, no variance, and no baseline configuration details. That alone would merit a request for revision, not a rejection.\n\nWhat's genuinely new: the integration of the Semantic Property Dependency Graph with multi-agent RL and LLM input generation in one tool, plus the header agent that learns to supply auth tokens from account-related operations in the spec. The header agent matters: it's a recent addition, not in the companion paper [11], and it's likely why AutoRestTest gets 2xx on OhSome while all four baselines get 0.\n\nThat last point is the soft spot that worries me. The paper never says whether OhSome's endpoints require authentication. If they do, and the baselines were run without equivalent credentials or header-generation support, the 26-vs-12 difference is an artifact of setup, not evidence that the SPDG+MARL+LLM architecture is better. The paper even calls the result 'initial, yet clear evidence' — that's overclaiming when the auth question is unaddressed. The same goes for the Spotify 5xx finding: it's a single observation, not a measured detection rate.\n\nOther soft spots are more minor. The Q-learning details (learning rate 0.1, discount 0.9) come from the authors' own ARAT-RL paper, which is fine, but there's no sensitivity analysis. The tool supports only OAS 3.0. The report examples are helpful. The paper is honest that the detailed approach lives in [11], so it reads as a tool demo with preliminary results, not a full research paper — and that's okay.\n\nMy bottom line: this is a solid artifact paper. The architecture is sensible, the repo is public, and the empirical question is concrete and testable. The right fix is to (a) state explicitly whether OhSome requires auth, (b) provide credentials/header logic to all baselines or justify why not, and (c) add multi-run stats with mean and variance. If those are in, the 26-vs-12 claim can be evaluated fairly. I'd send it to peer review — it deserves a serious referee — but I'd push for major revision on the evaluation before accepting the outperformance claim.","headline":"A real, coherent tool with a public artifact; the headline outperformance claim is plausible but under-supported, and the OhSome gap may be an auth confound.","tokens_in":6867,"tokens_out":2417,"would_cite":false,"duration_ms":23853,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoRestTest claims that combining a semantic dependency graph, five specialized Q-learning agents, and LLM-generated inputs covers more REST API operations in one hour than four state-of-the-art testing tools.","keywords":["REST API testing","multi-agent reinforcement learning","large language models","semantic property dependency graph","Q-learning","OpenAPI Specification","automated test generation"],"falsifier":"Run all five tools on the same four services on identical hardware and network, with the same time budget and comparable configuration effort, repeating each run several times and reporting variance; if any baseline reaches or exceeds 26 successfully processed operations, or matches AutoRestTest's 12 on OhSome when given the same LLM-based value generation, the central coverage claim fails.","tokens_in":5985,"feed_emoji":"🤖","tokens_out":4780,"duration_ms":45124,"temperature":0.7,"pith_summary":"This paper introduces AutoRestTest, a tool for automated REST API testing. It claims that AutoRestTest outperforms four state-of-the-art tools by integrating three mechanisms: a Semantic Property Dependency Graph, five specialized reinforcement-learning agents, and an LLM that generates realistic parameter and header values. In a one-hour evaluation on four real-world services, AutoRestTest covered 26 operations that returned 2xx responses, compared with 10 to 12 for the baselines, was the only tool to exercise the OhSome service successfully, and was the only one to detect a 5xx error on Spotify. The results are preliminary, but the claim is that combining dependency modeling, multi-agent learning, and LLM input generation is a more effective strategy than the individual approaches used by prior tools.","feed_headline":"AI agents cover 26 API operations, outdoing four tools","feed_subtitle":"A semantic graph, five Q-learning agents, and LLM inputs beat RESTler, EvoMaster, MoRest, and ARAT-RL on one-hour runs.","key_machinery":"The central object is the Semantic Property Dependency Graph (SPDG), a graph whose vertices are API operations and whose edges connect operations whose parameters, request bodies, or responses are semantically similar, measured with GloVe word embeddings above a threshold of 0.7 (or the top three matches when nothing exceeds the threshold). The dependency agent uses stored successful values to satisfy cross-operation dependencies. Five agents (operation, parameter, value, dependency, and header) each maintain Q-tables updated by Q-learning with epsilon-greedy action selection and a reward split: the operation agent is rewarded for 4xx and 5xx errors, while the other agents are rewarded for 2xx successes. The LLM supplies realistic values and headers. This machinery reduces the search space through semantic dependency edges and coordinates request-generation decisions across the five agents.","core_discovery":"The paper's central claim is that AutoRestTest achieves substantially higher operation coverage than four existing REST API testing tools by treating operation selection, parameter selection, value generation, dependency exploitation, and header construction as five separate learning problems coordinated through a shared Q-learning framework. The paper reports 26 unique successfully processed operations across FDIC, OMDb, OhSome, and Spotify in one-hour runs, versus 12 for ARAT-RL, 11 each for EvoMaster and MoRest, and 10 for RESTler. AutoRestTest returns 2xx responses for 12 operations on OhSome, where all baselines return only 4xx, and it is the only tool to produce a 5xx server error on Spotify. The authors interpret this as evidence that the SPDG's semantic edges let the dependency agent reuse outputs of one operation as inputs to another, while the LLM supplies realistic values that make otherwise unreachable operations succeed.","pith_inferences":["The reported gap may narrow if baseline tools receive equal LLM assistance or configuration tuning; a fair comparison would need identical hardware, network conditions, seeds, and repeated runs.","Because the operation agent is rewarded for 4xx and 5xx errors while other agents are rewarded for 2xx successes, the tool could favor operations that are easy to succeed at, so coverage count may not directly track fault-finding difficulty.","The SPDG construction depends only on semantic similarity between inputs and outputs, so the approach likely transfers to other interface description formats such as GraphQL schemas or gRPC service definitions.","An ablation removing the LLM, the SPDG, or individual agents would reveal which component drives the coverage gain; the paper does not provide such an ablation."],"forward_implications":["On the four evaluated services, AutoRestTest is the only one of the five tools that exercises OhSome operations successfully, suggesting the approach can handle services with heavy inter-operation dependencies.","The Spotify 5xx finding implies AutoRestTest can surface real server faults, not merely increase coverage counts.","Because the SPDG is built from any OAS 3.0 specification, the approach can in principle be applied to new REST APIs without per-service customization.","The Q-table outputs and cached LLM values make repeat executions cheaper and provide a record of which parameter combinations and dependencies led to success.","The reported numbers establish a new baseline for the one-hour successfully-processed-operations metric on these four services."],"supporting_citations":[{"why":"Baseline comparison tool and source of the default Q-learning learning rate and discount factor.","marker":"[5]"},{"why":"Baseline stateful REST API fuzzer against which AutoRestTest's coverage is compared.","marker":"[14]"},{"why":"Baseline evolutionary test generator included in the four-tool comparison.","marker":"[3]"},{"why":"Baseline model-based tool whose dynamic property graph is the closest dependency-modeling comparator.","marker":"[4]"},{"why":"Supplies the four real-world services (FDIC, OMDb, OhSome, Spotify) used in the evaluation.","marker":"[8]"},{"why":"Supports the choice of one-hour successfully processed operations as the comparison metric.","marker":"[15]"},{"why":"Companion paper containing the full description of SPDG construction and the multi-agent learning method.","marker":"[11]"}],"fun_headline_variants":["AutoRestTest: LLM+MARL covers 26 API ops, beats four tools","Semantic graph plus multi-agent RL boosts REST API testing","Five Q-learning agents + LLM beat four REST testing tools","REST API test tool more than doubles coverage with LLM","LLM and MARL tool finds 26 API ops, beats RESTler and EvoMaster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the four baseline tools were set up and run under conditions as favorable as AutoRestTest's, so that the one-hour count of 2xx operations measures tool quality rather than setup quality.","fun_headline_variants_meta":{"raw":{"variants":["AutoRestTest: LLM+MARL covers 26 API ops, beats four tools","Semantic graph plus multi-agent RL boosts REST API testing","Five Q-learning agents + LLM beat four REST testing tools","REST API test tool more than doubles coverage with LLM","LLM and MARL tool finds 26 API ops, beats RESTler and EvoMaster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000871,"raw_usage":{"total_tokens":3772,"prompt_tokens":949,"completion_tokens":2823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2725}},"tokens_in":565,"tokens_out":2823,"duration_ms":20558,"temperature":1.0,"reasoning_tokens":2725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:21:31.609028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all five tools on the same four services on identical hardware and network, with the same time budget and comparable configuration effort, repeating each run several times and reporting variance; if any baseline reaches or exceeds 26 successfully processed operations, or matches AutoRestTest's 12 on OhSome when given the same LLM-based value generation, the central coverage claim fails.","supporting_citations":[{"cited_title":"Adaptive rest api testing with rein- forcement learning,","cited_arxiv_id":null,"evidence_quote":"Baseline comparison tool and source of the default Q-learning learning rate and discount factor."},{"cited_title":"Restler: Stateful rest api fuzzing,","cited_arxiv_id":null,"evidence_quote":"Baseline stateful REST API fuzzer against which AutoRestTest's coverage is compared."},{"cited_title":"Restful api automated test case generation with evomaster,","cited_arxiv_id":null,"evidence_quote":"Baseline evolutionary test generator included in the four-tool comparison."},{"cited_title":"Morest: Model-based restful api testing with execution feedback,","cited_arxiv_id":null,"evidence_quote":"Baseline model-based tool whose dynamic property graph is the closest dependency-modeling comparator."},{"cited_title":"Enhancing rest api testing with nlp techniques,","cited_arxiv_id":null,"evidence_quote":"Supplies the four real-world services (FDIC, OMDb, OhSome, Spotify) used in the evaluation."},{"cited_title":"Testing restful apis: A survey,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of one-hour successfully processed operations as the comparison metric."}],"review_version":1}