REVIEW 4 major objections 4 minor 1 cited by
AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read AutoRestTest claims that combining a semantic dependency graph, five specialized Q-learning agents, and LLM-generated inputs covers more REST API operations in one hour than four state-of-the-art testing tools.
desk verdict A real, coherent tool with a public artifact; the headline outperformance claim is plausible but under-supported, and the OhSome gap may be an auth confound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Semantic Property Dependency Graph (SPDG), a graph whose vertices are API operations and whose edges connect operations whose parameters, request bodies, or responses are semantically similar, measured with GloVe word embeddings above a threshold of 0.7 (or the top three matches when nothing exceeds the threshold). The dependency agent uses stored successful values to satisfy cross-operation dependencies. Five agents (operation, parameter, value, dependency, and header) each maintain Q-tables updated by Q-learning with epsilon-greedy action selection and a reward split: the operation agent is rewarded for 4xx and 5xx errors, while the other agents are rewarded for 2xx successes. The LLM supplies realistic values and headers. This machinery reduces the search space through semantic dependency edges and coordinates request-generation decisions across the five agents.
What would settle it
Run all five tools on the same four services on identical hardware and network, with the same time budget and comparable configuration effort, repeating each run several times and reporting variance; if any baseline reaches or exceeds 26 successfully processed operations, or matches AutoRestTest's 12 on OhSome when given the same LLM-based value generation, the central coverage claim fails.
Extended reading notes
Core claim
The paper's central claim is that AutoRestTest achieves substantially higher operation coverage than four existing REST API testing tools by treating operation selection, parameter selection, value generation, dependency exploitation, and header construction as five separate learning problems coordinated through a shared Q-learning framework. The paper reports 26 unique successfully processed operations across FDIC, OMDb, OhSome, and Spotify in one-hour runs, versus 12 for ARAT-RL, 11 each for EvoMaster and MoRest, and 10 for RESTler. AutoRestTest returns 2xx responses for 12 operations on OhSome, where all baselines return only 4xx, and it is the only tool to produce a 5xx server error on Spotify. The authors interpret this as evidence that the SPDG's semantic edges let the dependency agent reuse outputs of one operation as inputs to another, while the LLM supplies realistic values that make otherwise unreachable operations succeed.
Load-bearing premise
The evaluation assumes that the four baseline tools were set up and run under conditions as favorable as AutoRestTest's, so that the one-hour count of 2xx operations measures tool quality rather than setup quality.
Editorial extensions
If this is right
- On the four evaluated services, AutoRestTest is the only one of the five tools that exercises OhSome operations successfully, suggesting the approach can handle services with heavy inter-operation dependencies.
- The Spotify 5xx finding implies AutoRestTest can surface real server faults, not merely increase coverage counts.
- Because the SPDG is built from any OAS 3.0 specification, the approach can in principle be applied to new REST APIs without per-service customization.
- The Q-table outputs and cached LLM values make repeat executions cheaper and provide a record of which parameter combinations and dependencies led to success.
- The reported numbers establish a new baseline for the one-hour successfully-processed-operations metric on these four services.
Reading between the lines
- The reported gap may narrow if baseline tools receive equal LLM assistance or configuration tuning; a fair comparison would need identical hardware, network conditions, seeds, and repeated runs.
- Because the operation agent is rewarded for 4xx and 5xx errors while other agents are rewarded for 2xx successes, the tool could favor operations that are easy to succeed at, so coverage count may not directly track fault-finding difficulty.
- The SPDG construction depends only on semantic similarity between inputs and outputs, so the approach likely transfers to other interface description formats such as GraphQL schemas or gRPC service definitions.
- An ablation removing the LLM, the SPDG, or individual agents would reveal which component drives the coverage gain; the paper does not provide such an ablation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents AutoRestTest, a tool for automated REST API testing that combines a Semantic Property Dependency Graph (SPDG), five specialized agents (operation, parameter, value, dependency, header), Q-learning, and LLM-based value and header generation. The authors evaluate AutoRestTest against RESTler, EvoMaster, ARAT-RL, and MoRest on four real-world services (FDIC, OMDb, OhSome, Spotify) and report that AutoRestTest successfully processed 26 unique operations versus 12, 11, 11, and 10 for the baselines, and that it was the only tool to obtain 2xx responses on OhSome and the only one to detect a 5xx error on Spotify. The paper also describes the tool's configuration options, CLI, and report generation. The reported results are described as preliminary.
Significance. If the reported results are reproducible and the differences are not due to configuration or stochastic variation, AutoRestTest would be a useful contribution to REST API testing, particularly in combining dependency modeling, multi-agent RL, and LLM-driven input generation. The artifact is publicly available, the tool addresses a practically relevant problem, and the evaluation uses four existing baseline tools and real-world services. However, the empirical evidence as presented is not yet sufficient to support the central claim of superiority: the comparison lacks run counts and variance, baseline configuration details are absent, and the OhSome result introduces a plausible authentication confound that is not addressed.
major comments (4)
- [Section IV, Table I] The headline superiority claim relies heavily on the OhSome service, where AutoRestTest reports 12 successful operations and all four baselines report 0. The manuscript does not state whether OhSome's endpoints require authentication, nor whether RESTler, EvoMaster, ARAT-RL, and MoRest were provided with equivalent credentials or header-generation facilities. Since Section II-B describes a Header Agent that supplies token authentication headers and a footnote states that this agent is a recent addition not described in the companion paper, the comparison is confounded unless authentication handling is demonstrated to be equal across tools. Please report the authentication requirements of each service and the exact authentication-related configuration supplied to each baseline.
- [Section IV] All effectiveness values appear to come from a single run per tool and service. No run counts, variance, confidence intervals, or statistical tests are reported. Given the use of epsilon-greedy Q-learning, random request mutation, and LLM sampling at temperature 0.7, there is substantial stochasticity in AutoRestTest; a lucky run could plausibly explain the observed differences. The paper should report at least several runs per tool and service with means and spread, and should state whether the one-hour window is wall-clock time and how LLM latency is accounted for.
- [Sections II-A and II-C] The components of AutoRestTest are not validated independently. The SPDG construction depends on a similarity threshold of 0.7 and a top-three fallback; the Q-learning parameters are set to 0.1 and 0.9 based on prior work; and the LLM engine is not specified for the experiments. No ablation is provided to isolate the contribution of the SPDG, the MARL agents, the LLM, or the header agent. As a result, even if the comparison against the baselines were sound, the paper would not support the claim that the proposed architecture as a whole is responsible for the observed effectiveness. Please add an ablation or at least controlled variants that disable individual components.
- [Section II-B, Footnote 1] The header agent is described only as leveraging 'account-related operations from the specification to supply basic token authentication headers,' and the footnote explicitly states that this agent is not described in the companion research paper. Because the header agent may be responsible for the OhSome successes and for the only detected Spotify 5xx error, its behavior is load-bearing for the evaluation. The paper should specify how account-related operations are identified, how tokens are obtained, and how the resulting headers are attached to requests, or it should point to a precise code location so that the evaluation is reproducible.
minor comments (4)
- [Abstract and Section II-B] The abstract and introduction describe 'five specialized agents,' but Section II-B says the REST agents 'consist of four specialized components' and does not count the dependency agent in that list. Please clarify the agent count and the placement of the dependency agent in the architecture.
- [Section III-A-2] The configuration section describes LLM temperature and engine selection but the experimental evaluation in Section IV does not state which LLM engine, model version, and temperature were used. This information is needed for reproducibility.
- [Listing 1] The example report in Listing 1 appears to contain keys with extra spacing, such as 'T i t l e' and 'D u r a t i o n'; if the actual tool output is valid JSON, please ensure the listing reproduces it faithfully.
- [References] Reference [6] is cited in the introduction as an example of reinforcement learning for REST API testing, but the text does not state its publication venue or year; please complete the bibliographic details in the citation if possible.
Circularity Check
No circular derivation: the paper's claims are empirical measurements against external services; the minor self-referential elements (benchmark sources, companion-paper citation) are not load-bearing.
full rationale
AutoRestTest's central claim is an observed experimental result: 26 successfully processed operations versus 12/11/11/10 for four baselines (Section IV, Table I). This is not a derivation from first principles, so there is no equation or fitted parameter whose output is, by construction, identical to its input. The Q-learning reward for other agents on 2xx responses (Section II-C.3) matches the evaluation metric of 2xx operation counts (Section IV), but that is an objective-function alignment, not a circular reduction; the tool's 2xx counts on external services (FDIC, OMDb, OhSome, Spotify) remain contingent empirical facts. The evaluation does rely on the authors' own prior work to select the services ('included in a recent study [8]') and the baseline lineup ('used in the ARAT-RL study [5]'), and the detailed method is deferred to the authors' companion paper [11]; these are self-references, but they do not carry the argument: the services are real public APIs and the baselines (RESTler, EvoMaster, MoRest, ARAT-RL) are independent implementations run for one hour. The footnote in Section II-B ('The header agent is a recent addition not described in our research paper') and the absence of baseline authentication configuration or variance data in Section IV are validity threats worth flagging, but they concern experimental fairness, not circularity of the claimed derivation. Accordingly the score reflects minor self-referential evaluation design, not a circular derivation chain.
Assumptions & free parameters
free parameters (4)
- SPDG edge similarity threshold =
0.7
- Q-learning learning rate =
0.1
- Q-learning discount factor =
0.9
- LLM temperature =
0.7
assumptions (4)
- domain assumption GloVe word embeddings provide semantically meaningful similarity between API parameter names, request bodies, and response fields.
- domain assumption Q-learning with epsilon-greedy action selection converges to a useful policy for API request generation.
- domain assumption The OpenAPI Specification is a complete and accurate description of the service under test.
- domain assumption OpenAI LLM APIs are available and generate realistic parameter values and headers.
invented entities (2)
-
Semantic Property Dependency Graph (SPDG)
-
Five REST agents (operation, parameter, value, dependency, header)
Cite this review
Pith. "Pith review of AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL." pith.science (2026). https://pith.science/paper/GWXE2PX7
@misc{pith2026250108600,
author = {Pith},
title = {Pith review of: AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL},
year = {2026},
howpublished = {\url{https://pith.science/paper/GWXE2PX7}},
note = {Machine review of arXiv:2501.08600}
}
read the original abstract
As REST APIs have become widespread in modern web services, comprehensive testing of these APIs is increasingly crucial. Because of the vast search space of operations, parameters, and parameter values, along with their dependencies and constraints, current testing tools often achieve low code coverage, resulting in suboptimal fault detection. To address this limitation, we present AutoRestTest, a novel tool that integrates the Semantic Property Dependency Graph (SPDG) with Multi-Agent Reinforcement Learning (MARL) and large language models (LLMs) for effective REST API testing. AutoRestTest determines operation-dependent parameters using the SPDG and employs five specialized agents (operation, parameter, value, dependency, and header) to identify dependencies of operations and generate operation sequences, parameter combinations, and values. Through an intuitive command-line interface, users can easily configure and monitor tests with successful operation count, unique server errors detected, and time elapsed. Upon completion, AutoRestTest generates a detailed report highlighting errors detected and operations exercised. In this paper, we introduce our tool and present preliminary findings, with a demonstration video available at https://www.youtube.com/watch?v=VVus2W8rap8.
Figures
Forward citations
Cited by 1 Pith paper
-
SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents
SAINT automatically generates both endpoint-level and scenario-based REST API tests for enterprise Java apps using static analysis and LLM agents, outperforming EvoMaster on code coverage in several benchmarks.
Reference graph
Works this paper leans on
-
[11]
A multi-agent approach for rest api testing with semantic graphs and llm-driven inputs,
M. Kim, T. Stennett, S. Sinha, and A. Orso, “A multi-agent approach for rest api testing with semantic graphs and llm-driven inputs,” arXiv preprint arXiv:2411.07098, 2024
arXiv 2024
-
[1]
L. Richardson, M. Amundsen, and S. Ruby, RESTful Web APIs: Services for a Changing World . O’Reilly Media, Inc., 2013
work page 2013
-
[2]
Automated test generation for rest apis: No time to rest yet,
M. Kim, Q. Xin, S. Sinha, and A. Orso, “Automated test generation for rest apis: No time to rest yet,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA
-
[3]
Restful api automated test case generation with evomaster,
A. Arcuri, “Restful api automated test case generation with evomaster,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 1, jan 2019
work page 2019
-
[4]
Morest: Model-based restful api testing with execution feedback,
Y . Liu, Y . Li, G. Deng, Y . Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, “Morest: Model-based restful api testing with execution feedback,” in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1406–1417
work page 2022
-
[5]
Adaptive rest api testing with rein- forcement learning,
M. Kim, S. Sinha, and A. Orso, “Adaptive rest api testing with rein- forcement learning,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . Los Alamitos, CA, USA: IEEE Computer Society, sep 2023, pp. 446–458
work page 2023
-
[6]
Deeprest: Automated test case generation for rest apis exploiting deep reinforcement learning,
D. Corradini, Z. Montolli, M. Pasqua, and M. Ceccato, “Deeprest: Automated test case generation for rest apis exploiting deep reinforcement learning,” 2024. [Online]. Available: https://arxiv.org/abs/ 2408.08594
arXiv 2024
-
[7]
Llamaresttest: Effective rest api testing with small language models,
M. Kim, S. Sinha, and A. Orso, “Llamaresttest: Effective rest api testing with small language models,” 2025. [Online]. Available: https://arxiv.org/abs/2501.08598
arXiv 2025
Show all 17 references
-
[8]
Enhancing rest api testing with nlp techniques,
M. Kim, D. Corradini, S. Sinha, A. Orso, M. Pasqua, R. Tzoref-Brill, and M. Ceccato, “Enhancing rest api testing with nlp techniques,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2023. New York, NY , USA: Associat...
2023
-
[9]
Leveraging large language models to improve rest api testing,
M. Kim, T. Stennett, D. Shah, S. Sinha, and A. Orso, “Leveraging large language models to improve rest api testing,” in Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , ser. ICSE-NIER’24. New York, NY , US...
2024
-
[10]
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: A Bradford Book, 2018
2018
-
[12]
Glove: Global vectors for word representation,
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014, pp. 1532–1543
2014
-
[13]
Value-decomposition networks for cooperative multi-agent learning,
P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel, “Value-decomposition networks for cooperative multi-agent learning,” arXiv preprint arXiv:1706.05296 , 2017
2017 arXiv
-
[14]
Restler: Stateful rest api fuzzing,
V . Atlidakis, P. Godefroid, and M. Polishchuk, “Restler: Stateful rest api fuzzing,” in Proceedings of the 41st International Conference on Software Engineering , ser. ICSE ’19. Piscataway, NJ, USA: IEEE Press, 2019, p. 748–758
2019
-
[15]
Testing restful apis: A survey,
A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing restful apis: A survey,” ACM Trans. Softw. Eng. Methodol. , aug 2023
2023
-
[16]
Experiment infrastructure, data, and results for autoresttest,
SE@GT, “Experiment infrastructure, data, and results for autoresttest,” https://github.com/selab-gatech/AutoRestTest, 2024
2024
-
[2022]
New York, NY , USA: Association for Computing Machinery, 2022, p. 289–301
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.