Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Automating a Complete Software Test Process Using LLMs: An Automotive Case Study

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A case study of truck-vehicle APIs shows that a complete manual test process, previously staffed by 2-3 full-time engineers, can be run end-to-end by a pipeline of large language models, with pass rates of 93-98% and perfect recall on…

desk verdict A credible industrial case study of full LLM-driven test automation, but the 'fully automatic substitution' claim needs repeated-run evidence it doesn't report. read the letter →

arxiv 2502.04008 v1 pith:465C7GM5 submitted 2025-02-06 cs.SE cs.AI

Shuai Wang , Yinan Yu , Robert Feldt , Dhasarathy Parthasarathy This is my paper · ORCID
classification cs.SEcs.AI
keywords softwaretestingvehicleAPItestautomationlargelanguagemodelRESTCANsignalsindustrialcasestudySPAPI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a real, largely manual industrial test process can be automated end-to-end by large language models, provided the process is decomposable and its steps demand judgment rather than creativity. The case is SPAPI, a web server in trucks whose REST APIs read and write vehicle states, and whose testing previously occupied 2-3 full-time engineers. The authors built a pipeline that preserves the manual workflow, assigns each step to an LLM, and reports that it generates valid test cases for 41 pre-verified APIs with a 93-98% pass rate, detects all 38 buggy APIs among 109 annotated ones with four false positives, and finds 22 developer-confirmed bugs in 193 newly developed industrial APIs. The broader claim is that the same recipe transfers to any similarly structured test process.

What carries the argument

The machinery is a four-stage pipeline that mirrors the manual process: template-based parsing with LLM assistance for documentation understanding; two LLM-driven matching stages that map API properties to CAN signals and then to Virtual Vehicle states, using DSPy typed signatures and Chain-of-Thought prompts to handle fuzzy names, abbreviations, informal pseudocode, unit mismatches, and one-to-many mappings; a Chain-of-Thought test-case generator that respects value ranges, units, and inter-parameter dependencies; and Jinja-template rendering of Pytest code that runs against the test rig and produces an auditable report. DSPy, a declarative LLM-pipeline library, supplies typed output validation and retries, which the authors use to stabilize the LLM responses across different models.

What would settle it

Run SPAPI-Tester's generated test suite against the same SPAPI server on production ECUs (or a hardware-in-the-loop harness) and compare each test's pass/fail verdict and detected bug list with the Virtual Vehicle rig results; any difference in verdict caused by signal timing, value ranges, or state semantics would show that the rig-based results do not transfer to the real vehicle.

Watch

Extended reading notes

Core claim

The central claim is that SPAPI testing—reading API specifications, retrieving CAN-signal and Virtual Vehicle documentation, fuzzy-matching inconsistent keys and values, generating and writing test cases, executing them, and triaging failures—can be fully automated by decomposing the process and replacing each manual step with an LLM invocation while keeping the original structure intact. On the authors' reported experiments, SPAPI-Tester achieved a 98% pass rate with two of four tested LLMs, all models exceeding 93%; test-case precision above 0.97 and recall up to 0.85 against expert-created ground truth; and detection of all 38 buggy APIs among 109 labeled APIs at 96% accuracy. On 193 new, unverified truck APIs the pipeline surfaced 23 failures, of which 22 were confirmed legitimate implementation bugs. The authors conclude that this manual process, previously staffed by 2-3 full-time engineers, has been effectively substituted by a fully automatic pipeline.

Load-bearing premise

The load-bearing premise is that the Virtual Vehicle test rig reproduces the real truck's electronic control-unit communication (CAN signals) faithfully enough that an API that passes or fails on the rig would pass or fail the same way on production hardware.

Editorial extensions

If this is right

  • SPAPI testing can proceed without human intervention, substituting a 2-3 FTE manual effort with an automatic pipeline and freeing testers for other work.
  • Because the process structure is preserved, the quality of the AI-driven automation can be assessed step by step, isolating which LLM step introduces errors.
  • The pipeline is largely model-agnostic: four LLMs achieved similar results without fine-tuning or retraining, so newer models can be swapped in with minimal adaptation.
  • From API specification to ready-to-run test cases takes about 11 seconds per API, compared with roughly two hours of expert effort for manual test creation.
  • On 193 new, unverified industrial APIs the pipeline found 22 implementation bugs confirmed by developers, showing it works on APIs it had not seen during development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same decomposition-and-LLM recipe should transfer to other gateway-style web servers and document-heavy, judgment-based test processes outside automotive; the authors hint at web-server testing but do not demonstrate it.
  • Beyond the paper: the recall loss of roughly 15 points when documentation omits units or types suggests that improving specification quality is a complementary, possibly cheaper, lever than further prompt engineering.
  • Beyond the paper: a hardware-in-the-loop replication of the study, running the generated tests against production ECUs, would quantify how much of the rig-based verdict transfers to real vehicles.
  • Beyond the paper: the LLM matching step that resolves spelling errors, abbreviations, and semantic equivalents could be reused for other integration tasks, such as merging legacy signal tables or diagnostics databases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an industrial case study in which a manual vehicle API testing process (SPAPI) is automated using LLMs. The authors decompose the process into documentation understanding, information matching, test-case generation, and execution/reporting, and present SPAPI-Tester, a DSPy-based pipeline. Experiments on 41 pre-verified APIs show pass rates between 0.93 and 0.98 depending on the LLM; on 12 expert-annotated APIs, precision exceeds 0.97 with recall between 0.73 and 0.85; on 109 expert-labeled APIs, all 38 buggy APIs were detected with 4 false positives; and on 193 newly developed APIs, 23 failures were confirmed by developers. The authors conclude that SPAPI testing, previously requiring 2-3 FTEs, has been effectively replaced by a fully automatic pipeline.

Significance. If the results hold, this is a significant demonstration that a judgment-heavy but well-structured industrial test process can be automated end-to-end with LLMs. The use of external human ground truth (expert test cases, expert bug labels, developer confirmations), the deployment on over 100 real industrial APIs, and the comparison across four LLMs are clear strengths. The recipe of preserving the process structure and using DSPy signatures is plausible and transferable. The paper is also honest about several limitations, such as missing units causing skipped attributes and the focus on a rig rather than a real vehicle. The main gap is that none of the reported results include repeated-run evidence, which is necessary to support the central 'fully automatic' claim.

major comments (3)
  1. [Section III.A-C and Tables II-IV] The paper does not report the number of independent runs behind any of the headline numbers. Since LLM invocations are stochastic, a single run can produce pass rates and failure detections that are not reproducible; DSPy's TypedPredictor and TypedChainOfThought retry only on format violations, not on semantic correctness of mappings or generated values. The claim in Section V that SPAPI testing has 'effectively been substituted by SPAPI-Tester, a fully automatic pipeline' requires evidence of run-to-run stability, e.g., repeated executions with seeds, reporting mean/range, and a flakiness rate. Without this, the reported 0.93-0.98 pass rates and the RQ4 detection results are compatible with a pipeline that intermittently fails or falsely flags APIs in recurring nightly runs, which would still require human triage.
  2. [Section IV.E (RQ4)] The 193-API deployment is presented as 'guaranteed to be unseen,' but the paper does not explain how this guarantee is established for the closed commercial LLMs used (GPT-3.5 and GPT-4o). No data-cutoff or training-data audit is provided, so one cannot exclude that these APIs or their documentation were part of pretraining. The authors should either remove the guarantee or provide evidence for it. This matters because RQ4 is the main evidence for real-world effectiveness, and the current phrasing overstates the novelty of the evaluation.
  3. [Section IV.B and Table III] The reported recall values (0.63-0.95) indicate that the pipeline skips attributes when documentation lacks units or variable types, and the paper states that all untested attributes are logged. The term 'complete automation' in Section V is therefore stronger than what the evaluation demonstrates: the pipeline fully automates the steps it executes, but it does not necessarily test all API attributes. The authors should report the fraction of attributes skipped and clarify whether 'full automation' means no human intervention in the executed steps or complete coverage of the specification.
minor comments (5)
  1. [Section II.A and Section V] The sentence that the case study 'focuses upon testing SPAPI in a rig, and not in the real vehicle' should be repeated in the abstract or conclusion, because the Section V claim of 'effectively substituted' could be misread as covering real-vehicle testing.
  2. [Section II.B.3] The phrase 'testers need to consider all test situations consider as many test situations as possible' appears to contain a duplicated clause and should be corrected.
  3. [Section IV.B] Please define the 'accuracy rate' precisely; from the numbers 38 true bugs and 4 false positives among 109 APIs, accuracy is (38 + (109-38-4))/109 = 0.963, so the reported 96% is consistent but the formula should be stated.
  4. [Table III] The recall averaging is ambiguous: the table reports per-category means but not whether they are weighted by number of APIs or properties; state how the 'Average' row is computed.
  5. [References] Reference [4] duplicates reference [3] (same title, authors, and venue); use a single entry to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation rests on external expert ground truth and developer-confirmed bug reports, not on the system's own outputs.

full rationale

This paper is an empirical industrial case study rather than a derivation chain, and no load-bearing step reduces to its own inputs. SPAPI-Tester's outputs (matched objects, generated test cases, pass/fail results) are evaluated against independent, human-created references: expert ground-truth test cases for 12 APIs (Table III), expert bug/non-bug labels for 109 APIs (Section IV.B), and developer confirmation of the 23 failures found on 193 newly developed APIs (Section IV.E). These are external benchmarks, not quantities defined in terms of the pipeline's own outputs. The Section V claim that SPAPI testing 'has effectively been substituted by SPAPI-Tester, a fully automatic pipeline' is a summary of those externally validated results rather than a definitional consequence. The only self-citations are to prior LLM-testing work in the related-work and background discussion; they are not load-bearing, do not invoke a uniqueness theorem, and do not justify the central premise. The skeptic's concern about missing repeated-run evidence is a validity and robustness threat, not circularity: it questions whether the reported pass rates and failure detections generalize across stochastic runs, but it does not show that any claimed result was derived from the claim itself. No circular step can be quoted because no step equates a prediction to a fitted input or imports a conclusion from the authors' own prior work by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim does not depend on fitted numeric parameters or newly postulated entities. It rests on domain assumptions about the fidelity of the test rig, the correctness of expert labels, the representativeness of the documented inconsistency categories, and the stability of single-run LLM evaluations. The main burden is therefore empirical and contextual, not mathematical.

assumptions (4)
  • domain assumption The Virtual Vehicle (VV) test rig faithfully emulates real in-vehicle states via the same CAN signals used in production, so test results obtained on the rig transfer to the real vehicle.
    Section II.A limits the case study to a rig with VV emulating the superset of vehicle states; all pass-rate, coverage, and failure-detection numbers in Section IV are measured against this emulated environment.
  • domain assumption The expert-labeled ground truth (12 APIs for coverage, 109 APIs for buggy/non-buggy labels) and developer confirmations of the 193-API failures correctly reflect expected API behavior.
    Sections IV.B and IV.E use these human labels as the evaluation oracle; incorrect or biased labels would invalidate the reported precision, recall, and failure-detection accuracy.
  • domain assumption The five documented inconsistency categories (Table I) plus informal pseudocode and unit issues are representative of the full SPAPI documentation space.
    Section II.C defines these as the core obstacles to automation, and RQ2 evaluates only these categories; if the real documentation contains other failure modes, the claimed generality does not follow.
  • domain assumption A single run of each LLM on each API sample is representative of that LLM's performance in the pipeline.
    Section IV reports only point estimates for pass rate, precision, recall, and F1; no repeated runs, seeds, or variance measures are given, so the numbers assume output stability despite LLM sampling stochasticity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automating a Complete Software Test Process Using LLMs: An Automotive Case Study." pith.science (2026). https://pith.science/paper/465C7GM5

@misc{pith2026250204008,
  author       = {Pith},
  title        = {Pith review of: Automating a Complete Software Test Process Using LLMs: An Automotive Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/465C7GM5}},
  note         = {Machine review of arXiv:2502.04008}
}
read the original abstract

Vehicle API testing verifies whether the interactions between a vehicle's internal systems and external applications meet expectations, ensuring that users can access and control various vehicle functions and data. However, this task is inherently complex, requiring the alignment and coordination of API systems, communication protocols, and even vehicle simulation systems to develop valid test cases. In practical industrial scenarios, inconsistencies, ambiguities, and interdependencies across various documents and system specifications pose significant challenges. This paper presents a system designed for the automated testing of in-vehicle APIs. By clearly defining and segmenting the testing process, we enable Large Language Models (LLMs) to focus on specific tasks, ensuring a stable and controlled testing workflow. Experiments conducted on over 100 APIs demonstrate that our system effectively automates vehicle API testing. The results also confirm that LLMs can efficiently handle mundane tasks requiring human judgment, making them suitable for complete automation in similar industrial contexts.

Figures

Figures reproduced from arXiv: 2502.04008 by the authors.

Figure 1
Figure 1. We present the case of automatically testing SPAPI, an in-vehicle web [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A comparative illustration of the SPAPI architecture – (1) a web server [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Three tiers of SPAPI operation (1) presentation - SPAPI objects, (2) [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The process of setting and getting vehicle status according to the API [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Architecture and workflow of SPAPI-Tester: The pipeline largely preserves the manual process and selectively uses LLMs to automate discrete steps. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: A DSPy Signature for automating API to CAN lookup (simplified). [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Templatized examples for guiding API to CAN look up. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Chain-of-Thought prompt for test case generation (simplified). [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Matching performance on informal pseudocoded mappings. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 35 canonical work pages

  1. [1]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSE-FoSE 2023, Melbourne, Australia, May 14-20, 2023 , pp. 31–53, IEEE, 2023

    work page 2023

  2. [2]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. C. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” CoRR, vol. abs/2308.10620, 2023

    arXiv 2023

  3. [3]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Trans. Software Eng., vol. 50, no. 4, pp. 911–936, 2024

    work page 2024

  4. [4]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , 2024

    2024

  5. [5]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” arXiv preprint arXiv:2304.05128 , 2023

    arXiv 2023

  6. [6]

    No more manual tests? evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y . Lou, M. Liu, S. Ding, K. Wang, Y . Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” arXiv preprint arXiv:2305.04207 , 2023

    arXiv 2023

  7. [7]

    Enhancing software development practices with ai insights in high-tech companies,

    D. Ajiga, P. A. Okeleke, S. O. Folorunsho, and C. Ezeigweneme, “Enhancing software development practices with ai insights in high-tech companies,” 2024

    work page 2024

  8. [8]

    Intent-driven mobile gui testing with autonomous large language model agents,

    J. Yoon, R. Feldt, and S. Yoo, “Intent-driven mobile gui testing with autonomous large language model agents,” in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST) , pp. 129–139, IEEE, 2024

    work page 2024

Show all 42 references
  1. [9]

    Towards autonomous test- ing agents via conversational large language models,

    R. Feldt, S. Kang, J. Yoon, and S. Yoo, “Towards autonomous test- ing agents via conversational large language models,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE), pp. 1688–1693, IEEE, 2023

    2023

  2. [10]

    Llms and process mining: Challenges in rpa: Task grouping, labelling and connector recommen- dation,

    M. Fani Sani, M. Sroka, and A. Burattin, “Llms and process mining: Challenges in rpa: Task grouping, labelling and connector recommen- dation,” in International Conference on Process Mining , pp. 379–391, Springer, 2023

    2023

  3. [11]

    Llms for intelligent software testing: a comparative study,

    M. Boukhlif, N. Kharmoum, and M. Hanine, “Llms for intelligent software testing: a comparative study,” in Proceedings of the 7th Inter- national Conference on Networking, Intelligent Systems and Security , pp. 1–8, 2024

    2024

  4. [12]

    Testing restful apis: A survey,

    A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing restful apis: A survey,” ACM Trans. Softw. Eng. Methodol. , vol. 33, nov 2023

    2023

  5. [13]

    Modeling 3-tiered web applications,

    X. Liu, J. Heo, and L. Sha, “Modeling 3-tiered web applications,” in 13th IEEE international symposium on modeling, analysis, and simulation of computer and telecommunication systems , pp. 307–310, IEEE, 2005

    2005

  6. [14]

    Openapi standard,

    OpenAPI, “Openapi standard,” 2023. https://www.openapis.org

    2023

  7. [15]

    Openapi template,

    OpenAPI, “Openapi template,” 2024. https://openapi-generator.tech/docs /templating

    2024

  8. [16]

    Dspy: Compiling declarative language model calls into self-improving pipelines,

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, et al. , “Dspy: Compiling declarative language model calls into self-improving pipelines,” arXiv preprint arXiv:2310.03714 , 2023

    arXiv

  9. [17]

    Introducing structured outputs in the api

    OpenAI, “Introducing structured outputs in the api.” https://openai.com/ index/introducing-structured-outputs-in-the-api/, 2023. Accessed: 2024- 10-21

    2023

  10. [18]

    Fundamentals of creep, testing methods and development of test rig for the full-scale crossarm: A review,

    M. Asyraf, M. Ishak, M. Razman, and M. Chandrasekar, “Fundamentals of creep, testing methods and development of test rig for the full-scale crossarm: A review,” Jurnal Teknologi, vol. 81, no. 4, 2019

    2019

  11. [19]

    Testing restful apis: A survey,

    A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing restful apis: A survey,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 1, pp. 1–41, 2023

    2023

  12. [20]

    Resource-based test case generation for restful web services,

    M. Zhang, B. Marculescu, and A. Arcuri, “Resource-based test case generation for restful web services,” in Proceedings of the genetic and evolutionary computation conference , pp. 1426–1434, 2019

    2019

  13. [21]

    Resource and dependency based test case generation for restful web services,

    M. Zhang, B. Marculescu, and A. Arcuri, “Resource and dependency based test case generation for restful web services,” Empirical Software Engineering, vol. 26, no. 4, p. 76, 2021

    2021

  14. [22]

    Automated black-and white-box testing of restful apis with evomaster,

    A. Arcuri, “Automated black-and white-box testing of restful apis with evomaster,” IEEE Software, vol. 38, no. 3, pp. 72–78, 2020

    2020

  15. [24]

    Restful api automated test case generation with evomaster,

    A. Arcuri, “Restful api automated test case generation with evomaster,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 28, no. 1, pp. 1–37, 2019

    2019

  16. [25]

    Javascript instrumentation for search-based software testing: A study with restful apis,

    M. Zhang, A. Belhadi, and A. Arcuri, “Javascript instrumentation for search-based software testing: A study with restful apis,” in 2022 IEEE Conference on Software Testing, Verification and Validation (ICST) , pp. 105–115, IEEE, 2022

    2022

  17. [26]

    Model-based testing of breaking changes in node. js libraries,

    A. Møller and M. T. Torp, “Model-based testing of breaking changes in node. js libraries,” in Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering , pp. 409–419, 2019

    2019

  18. [27]

    Pythia: grammar-based fuzzing of rest apis with coverage-guided feed- back and learning-based mutations,

    V . Atlidakis, R. Geambasu, P. Godefroid, M. Polishchuk, and B. Ray, “Pythia: grammar-based fuzzing of rest apis with coverage-guided feed- back and learning-based mutations,” arXiv preprint arXiv:2005.11498 , 2020

    2005 arXiv

  19. [28]

    Resttestgen: automated black-box testing of restful apis,

    E. Viglianisi, M. Dallago, and M. Ceccato, “Resttestgen: automated black-box testing of restful apis,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) , pp. 142–152, IEEE, 2020

    2020

  20. [29]

    Jsongen: A quickcheck based library for testing json web services,

    C. Benac Earle, L.- ˚A. Fredlund, ´A. Herranz, and J. Mari ˜no, “Jsongen: A quickcheck based library for testing json web services,” in Proceedings of the Thirteenth ACM SIGPLAN workshop on Erlang , pp. 33–41, 2014

    2014

  21. [30]

    Test-the-rest: An approach to testing restful web-services,

    S. K. Chakrabarti and P. Kumar, “Test-the-rest: An approach to testing restful web-services,” in 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns , pp. 302– 308, IEEE, 2009

    2009

  22. [31]

    Model-driven testing of restful apis,

    T. Fertig and P. Braun, “Model-driven testing of restful apis,” in Proceedings of the 24th International Conference on World Wide Web , pp. 1497–1502, 2015

    2015

  23. [32]

    Test suite generation with the many independent objec- tive (mio) algorithm,

    A. Arcuri, “Test suite generation with the many independent objec- tive (mio) algorithm,” Information and Software Technology , vol. 104, pp. 195–206, 2018

    2018

  24. [33]

    Intelligent rest api data fuzzing,

    P. Godefroid, B.-Y . Huang, and M. Polishchuk, “Intelligent rest api data fuzzing,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp. 725–736, 2020

    2020

  25. [34]

    Improving test case generation for rest apis through hierarchical clustering,

    D. Stallenberg, M. Olsthoorn, and A. Panichella, “Improving test case generation for rest apis through hierarchical clustering,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE), pp. 117–128, IEEE, 2021

    2021

  26. [35]

    for- est: A tree-based approach for fuzzing restful apis,

    J. Lin, T. Li, Y . Chen, G. Wei, J. Lin, S. Zhang, and H. Xu, “for- est: A tree-based approach for fuzzing restful apis,” arXiv preprint arXiv:2203.02906, 2022

    2022 arXiv

  27. [36]

    Restest: Black- box constraint-based testing of restful web apis,

    A. Martin-Lopez, S. Segura, and A. Ruiz-Cort ´es, “Restest: Black- box constraint-based testing of restful web apis,” in Service-Oriented Computing: 18th International Conference, ICSOC 2020, Dubai, United Arab Emirates, December 14–17, 2020, Proceedings 18 , pp. 459–475, Spri...

    2020

  28. [37]

    Application of api automation test- ing based on microservice mode in industry software,

    N. Li, J. Wang, C. Chen, and H. Hu, “Application of api automation test- ing based on microservice mode in industry software,” in Proceedings of the International Conference on Algorithms, Software Engineering, and Network Security , pp. 460–464, 2024

    2024

  29. [38]

    Optimizing microservices and api testing pipelines with ai,

    T. Olasehinde and S. Shekhar, “Optimizing microservices and api testing pipelines with ai,”

  30. [39]

    Leveraging large language models to improve rest api testing,

    M. Kim, T. Stennett, D. Shah, S. Sinha, and A. Orso, “Leveraging large language models to improve rest api testing,” in Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , pp. 37–41, 2024

    2024

  31. [40]

    Kat: Dependency-aware automated api testing with large language models,

    T. Le, T. Tran, D. Cao, V . Le, T. N. Nguyen, and V . Nguyen, “Kat: Dependency-aware automated api testing with large language models,” in 2024 IEEE Conference on Software Testing, Verification and Valida- tion (ICST), pp. 82–92, IEEE, 2024

    2024

  32. [41]

    Automating rest api postman test cases using llm,

    S. Deepika Sri, M. Aadil S, S. Varshini R, R. CSP Raman, G. Rajagopal, and S. Taranath Chan, “Automating rest api postman test cases using llm,” arXiv e-prints, pp. arXiv–2404, 2024

    2024

  33. [42]

    You can rest now: Automated specification inference and black-box testing of restful apis with large language models,

    A. Decrop, G. Perrouin, M. Papadakis, X. Devroey, and P.-Y . Schobbens, “You can rest now: Automated specification inference and black-box testing of restful apis with large language models,” arXiv preprint arXiv:2402.05102, 2024

    2024 arXiv

  34. [43]

    Apitestgenie: Automated api test generation through generative ai,

    A. Pereira, B. Lima, and J. P. Faria, “Apitestgenie: Automated api test generation through generative ai,” arXiv preprint arXiv:2409.03838, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.