Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

Dengcheng He; Haotian Si; Jianqiang Zhao; Jiate Li; Junyi Chen; Meng Li; Ruifeng Nie; Shuyang Yu

REVIEW 4 major objections 4 minor 11 references

A new metric, Skill Test Coverage, measures how completely a cloud skill's test suite covers its specified operational behaviors, and deployment on 157 production skills found 36.3% initially fall below the mandatory 80% gate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:02 UTC pith:M76NGAD5

load-bearing objection A useful, well-scoped new coverage metric for cloud skills with real deployment data, but the headline numbers rest on an unvalidated LLM+human pipeline. the 4 major comments →

arxiv 2607.22015 v1 pith:M76NGAD5 submitted 2026-07-24 cs.SE

Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

Haotian Si , Junyi Chen , Shuyang Yu , Ruifeng Nie , Jiate Li , Jianqiang Zhao , Meng Li , Dengcheng He This is my paper

classification cs.SE

keywords Cloud SkillsSkill Test CoverageTest ObligationsAI AgentsSoftware TestingTest AdequacyNatural-language specificationIndustrial deployment

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that passing a cloud skill's test cases does not reveal which specified behaviors were never tested. It defines Skill Test Coverage: the fraction of operational test obligations (discrete, observable behaviors such as creating a VPC or handling a failure) covered by at least one test case. The authors build a pipeline that recovers these obligations from natural-language skill packages using independent LLM runs and human review, then maps each test case to the obligations it exercises using the prompt and initial resource state. Deployed as a mandatory release gate on 157 production skills, the measurement shows 57 skills (36.3%) initially fall below the 80% threshold, and the generated reports offer concrete, source-grounded recommendations for closing gaps. If the measurement is sound, it gives cloud platforms an auditable, obligation-level way to see untested behaviors before release.

Core claim

The paper's central claim is that Skill Test Coverage — the ratio of operational test obligations covered by at least one testcase to the total reviewed obligation inventory — can be measured in practice for workflow-oriented cloud skills, despite the implicit, natural-language nature of the coverage units. The paper formalizes obligations as normalized records of observable operations with activation conditions and expected results, defines coverage as a specification-level verification condition checked for each testcase scenario, and implements the measurement through a pipeline of parallel LLM candidate generation, human review, deterministic aggregation, and source-grounded gap recommen

What carries the argument

The central object is the operational test obligation O(S), a normalized record ⟨operation, guard, expected result, source span⟩ representing a release-relevant behavior entailed by the Skill's workflow semantics. Coverage is defined per testcase via the verification condition S,Γ(t)⊨ô: every conforming execution of the Skill compatible with the testcase's prompt and initial resource state must realize the obligation. The testcase–obligation matrix M_ij accumulates these judgments; suite coverage is the fraction of columns with at least one 1. The measurement pipeline pairs three independent agent runs that propose obligation inventories with human review to fix the denominator, and per-test

Load-bearing premise

The entire measurement rests on the assumption that the reviewed obligation inventory and per-testcase covered/uncovered labels, produced by LLM agents and corrected by human reviewers, accurately capture which behaviors the skill actually specifies and which of those each test case would exercise; if reviewers systematically miss low-salience obligations or mislabel coverage, the reported percentages are unreliable.

What would settle it

Take a sample of skills from the deployment, have independent expert teams (without access to the original review records) produce obligation inventories and coverage labels from the same skill packages and test cases, then measure agreement. If inter-team agreement on obligation sets or on coverage status of matched obligations is low, or if known deliberately inserted untested behaviors are not flagged, the measurement claim fails. Alternatively, instrument a sandbox environment to record actual operations during testcase execution and check whether operations labeled 'covered' are indeed ex

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Cloud platforms can enforce a coverage gate before task-success evaluation, preventing high success on narrow scenarios from bypassing test-adequacy requirements.
A third of in-development skills at the deploying platform initially fail the mandatory 80% gate, so untested behaviors are not rare at submission.
Obligation-level gap reports translate into source-grounded recommendations: add a testcase, clarify prompt or resource state, revise ambiguous skill text, or record an explicit exception.
The metric is monotonic under test addition and complete with respect to the reviewed obligation criterion, so re-measurement after remediation gives a clear progress signal.
The authors state the deployment records do not yet establish recommendation acceptance, post-remediation gains, annotation reliability, or generalization beyond the deploying platform.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same obligation-recovery and coverage-mapping strategy could transfer to other natural-language operational artifacts, such as runbooks, on-call playbooks, or API integration guides, where coverage units are equally implicit.
If coupled with fault-injection or trace-based validation, the coverage labels could be checked against actual executions, providing an independent test of the measurement itself.
The lack of inter-rater agreement and external ground truth means the 36.3% figure should be read as an internal estimate; a reproducibility study with independent expert teams on a sample of skills would sharpen or refute the headline percentage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful, well-scoped new coverage metric for cloud skills with real deployment data, but the headline numbers rest on an unvalidated LLM+human pipeline.

read the letter

First, the honest take: this is a genuinely useful new QA instrument for a fast-growing artifact class. The paper defines Skill Test Coverage for workflow-oriented Cloud Skills, deploys it as a release gate on 157 production skills at Alibaba Cloud, and shows a third of them initially fail the mandatory 80% bar. That is new, it is real industrial data, and the formalization is unusually careful for an applied SE paper. The coverage unit (operational test obligation) is well-defined as a guarded behavior record, the coverage relation uses universal quantification over conforming executions, and the paper states its claim boundary explicitly: coverage is an adequacy signal, not correctness. The pipeline — parallel LLM candidate generation, disagreement-preserving aggregation, human review, deterministic aggregation — is a credible engineering solution to the problem that the denominator and the testcase-to-obligation links are implicit in natural-language packages. The recommendation module that turns confirmed gaps into source-grounded actions is also a nice practical touch.

The soft spot is the one flagged in the stress test: the formal metric and the measured numbers are not the same object. Section 3.1 says plainly that the pipeline does not enumerate L(S) or prove the verification conditions; it substitutes LLM+human judgments for the universal quantification. That is a defensible engineering approximation, but it means the 36.3% failure rate is only as reliable as those judgments. There is no inter-rater reliability, no external oracle, and no error bars. The authors acknowledge this in Section 6 — they list reviewer agreement as future work — which is honest, but it does limit what the paper can claim. The same is true for the threshold: 80% is release policy, not a derived bound. None of this is fatal. The central argument holds up: untested behaviors are common, and obligation-level reporting gives actionable evidence beyond a scalar score. The paper simply overreaches slightly in the abstract when it frames the deployment numbers as established fact.

This paper is for anyone building evaluation pipelines for agent skills or thinking about test adequacy for natural-language artifacts. It deserves serious peer review; the formal core is solid, the deployment is real, and the weaknesses are addressable with a reliability study and artifact release.

Referee Report

4 major / 4 minor

Summary. The paper introduces Skill Test Coverage (STC), a metric for how completely a Cloud Skill's test suite covers the operational behaviors, or obligations, specified by the natural-language Skill package. It formalizes obligations, a testcase–obligation coverage relation, and a suite-level coverage measure. Because obligations and coverage links are not explicit in Skill packages, the paper builds a measurement pipeline that uses parallel LLM agents to propose obligation inventories and per-testcase coverage labels, followed by human review. The pipeline was deployed as a mandatory release gate at Alibaba Cloud. In an analysis of 157 initial coverage measurements, 57 (36.3%) fall below the 80% release threshold and 76 (48.4%) below the recommended 90% level; 132 recommendation reports contain 639 source-grounded suggestions. The paper's central claim is that incomplete tests are common before gate-driven remediation and that obligation-level analysis provides actionable evidence beyond a scalar score.

Significance. If the measurement is reliable, this is a useful contribution: it gives cloud platforms an auditable, obligation-level way to expose untested behaviors in agent skills, complementing task-success evaluation. The formal definition is explicit and has desirable sanity properties (boundedness, monotonicity, completeness relative to the selected obligation criterion). The paper also reports a real deployment, which is valuable. However, the headline quantitative claims rest on a pipeline that substitutes reviewed LLM judgments for the formal verification judgments in Section 2.3, and the paper provides no reliability evidence—no inter-rater agreement, no external oracle study, and no uncertainty quantification on any reported percentage. The significance is therefore conditional on additional validation or on a careful reframing of what is being claimed.

major comments (4)

[§3.1 vs §2.3] The formal coverage definition in Section 2.3 requires checking the verification condition S,Γ(t)|=^o for every conforming execution, plus decision-completeness, which makes M_ij well-defined. Section 3.1 explicitly says the pipeline does not enumerate L(S) or prove these verification conditions; instead, agents propose and reviewers correct labels. No step checks decision-completeness: Section 3.4's input contract only requires consistency and satisfiability, not the '|{match(o,σ,t)|σ∈L_S(t)}|=1' condition. Consequently, M_ij can be set arbitrarily when a testcase's scenario does not uniquely determine whether an obligation is realized. This is load-bearing because the reported 36.3% failure rate is computed from these M_ij values. Please either implement or automate a decision-completeness check for the actual pipeline, or restrict the paper's claims to 'reviewed coverage labels' and p
[§2.2, §3.2–3.4, §6] The obligation inventory O(S) is defined by a semantic completeness and non-redundancy condition: every release-relevant clause must map to exactly one obligation. The pipeline obtains this inventory from three LLM runs and human review. The paper provides no inter-rater reliability for the review step—indeed, Section 6 lists 'evaluate reviewer agreement' as future work—and no comparison against an external oracle or seeded ground truth. If reviewers or the LLM systematically miss low-salience obligations, the denominator shrinks and STC is overestimated; if they over-include supporting material, STC is underestimated. The 36.3% figure therefore has no quantified measurement error. Please add a reliability study (e.g., multiple reviewers on a sample of Skills, agreement metrics, and a sensitivity analysis on obligation additions/deletions) or explicitly downgrade the empirical claims to
[§4.2, Figure 3] The headline statistics—mean 80.3%, median 91.3%, IQR 68–100%, 36.3% below 80%—are point estimates from one pass through the pipeline. There are no confidence intervals or error bars, so the reader cannot tell whether the 36.3% is, say, 30% or 43% under reasonable variation in reviewer judgment. Given that the paper's central message is that 'incomplete tests are common before gate-driven remediation,' this uncertainty should be quantified. If the authors prefer not to add a formal reliability study, they should at least provide a bootstrap or sensitivity analysis over reviewer noise and state the resulting ranges.
[§2.4, §4.1] The 80% and 90% thresholds are introduced as release policy, and Section 2.4 correctly notes they are not theoretical sufficiency bounds. That framing is appropriate. However, the conclusion that 'incomplete tests are common' depends on these thresholds: with a 50% threshold the failure rate would be much smaller, and with a 95% threshold nearly all Skills would fail. The paper should either justify the thresholds empirically or present the coverage distribution more neutrally, so that the reader can evaluate the claim independently of the chosen cutoffs.

minor comments (4)

[Figure 3] The axis labels are confusing: the horizontal axis is 'Cumulative proportion of Skills,' but the vertical axis also shows cumulative distribution values. Consider labeling the horizontal axis as 'rank percentile' or redrawing as an empirical CDF with explicit step lines.
[§4.2] The recommendation workload (639 recommendations, median 4 per Skill) is reported without a breakdown by action type (add testcase, clarify prompt, revise Skill, or exception). A small table or sentence categorizing these would make the 'actionable' claim easier to assess.
[§3.2] The three agents are said to be 'redundancy rather than votes,' and the union is retained. This is a sensible design, but the paper does not report how often the three runs actually disagreed or how often reviewers changed the union. Such numbers would help calibrate the need for review.
[§4.1] The sentence 'Since this process went online, the platform has evaluated 157 Cloud Skills' would benefit from a date range and a description of how Skills were selected to enter the pipeline (e.g., all submissions or a sample). This would clarify potential selection bias.

Circularity Check

0 steps flagged

No significant circularity: STC is a defined metric measured by a review pipeline; the 36.3% figure is an empirical observation, not an input or a fitted prediction.

full rationale

The paper's derivation chain is definitional and empirical rather than circular. STC(S,T) is explicitly defined as the fraction of reviewed obligations covered by at least one testcase: STC(S,T) = (1/n) Σ c_j, with c_j = OR_i M_ij (Section 2.3). The coverage units O(S) and matrix labels M_ij are established by a pipeline of independent LLM candidate generation plus human review (Sections 3.2–3.4). The central quantitative claim—57 of 157 (36.3%) initial measurements fall below the 80% gate—is a descriptive statistic over those reviewed measurements, not a quantity derived from the definition by construction, nor a parameter fitted to data and then renamed as a prediction. The 80% threshold is explicitly a release policy, not a theoretical sufficiency bound (Section 2.4). The pipeline is not used to predict a held-out quantity; it is the measurement itself. The paper contains no load-bearing self-citations: all cited works are external, including the SKILL.md study [4], and no uniqueness theorem or ansatz is imported from the authors' prior work. The acknowledged limitations—no inter-rater agreement study, no external ground truth, no post-remediation tracking (Sections 4.2 and 6)—are reliability and external-validity concerns, not circularity. An unreliable measurement instrument does not make the reported reading an input to its own derivation. Thus, while the 36.3% figure should be interpreted cautiously, it does not reduce to the paper's own definitions or citations.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The central definition is self-contained, but the empirical deployment rests on several domain assumptions: natural-language Skills can be modeled as trace languages; obligations can be reliably extracted by LLM and reviewers; and the chosen thresholds are valid. The paper provides no external validation for these assumptions.

free parameters (2)

80% release gate threshold = 80%
Hand-chosen mandatory release policy; the headline statistic '36.3% below 80%' depends on this value, which is not derived from theory or risk data. Stated in §4.1.
90% recommended coverage level = 90%
Hand-chosen recommended target, also not derived; the statement '48.4% below 90%' depends on it. Stated in §4.1.

axioms (5)

domain assumption Conforming executions of a Cloud Skill are finite traces of observable operations and states.
Section 2.1 defines L(S) this way; natural-language Skills are not executable specifications, so this is an idealization.
domain assumption The Skill package semantically entails a well-defined set of behavior clauses B(S), and semantic operation identity ≃ is decidable by reviewers.
Section 2.2 uses B(S) and ≃; Table 1 acknowledges natural-language ambiguity, but the formal definitions assume a well-defined domain.
ad hoc to paper The release-relevant criteria (workflow, prescriptive, triggerable, observable, testable) produce a valid coverage denominator.
Section 2.2 defines R(S) by these criteria; there is no external evidence that this boundary matches what should be tested.
ad hoc to paper The LLM agents and reviewers accurately recover obligations and coverage labels after review.
Sections 3.2–3.4 rely on Qwen3.7-Max and expert review; no accuracy or inter-rater reliability data are provided.
domain assumption Testcases entering coverage review satisfy the well-formedness and decision-completeness conditions of Section 2.3.
Section 3.4 checks only the input contract of Section 2.1; decision-completeness is not verified, though cov_S is defined only for such testcases.

invented entities (1)

Operational test obligation no independent evidence
purpose: The coverage unit for measuring how completely a Skill's test suite exercises its specified behaviors.
A conceptual construct introduced by this paper and defined via the semantic domain R(S); there is no external benchmark showing these obligations correspond to real, independently verifiable test requirements.

pith-pipeline@v1.3.0-alltime-deepseek · 10145 in / 14815 out tokens · 139232 ms · 2026-08-01T06:02:59.617890+00:00 · methodology

0 comments

read the original abstract

Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources. Workflow-oriented Skills specify multiple resource operations, user choices, validation steps, and recovery behavior. Existing Skill evaluation primarily measures whether a Skill improves task success. Passing the available testcases, however, does not reveal which behaviors specified by the Skill have never been tested. This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations. We define the coverage units, testcase coverage relation, calculation procedure, and claim boundary. Because these units and links are not explicit in natural-language Skill packages, we develop a measurement pipeline that recovers operational obligations, organizes their workflow context, maps testcases using both user prompts and initial resource states, and produces an auditable coverage report. Model-assisted candidate generation is combined with expert review to ensure that the reported coverage remains grounded in the original Skill and testcase evidence. A post-review module then translates confirmed coverage gaps into source-grounded test-improvement recommendations.

Figures

Figures reproduced from arXiv: 2607.22015 by Dengcheng He, Haotian Si, Jianqiang Zhao, Jiate Li, Junyi Chen, Meng Li, Ruifeng Nie, Shuyang Yu.

**Figure 1.** Figure 1: Code Coverage and Skill Test Coverage. Code exposes executable lines and branch outcomes as coverage units [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: The Skill Test Coverage measurement platform. Independent agents propose obligation inventories under the same [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Ranked distribution of 157 initial coverage mea [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Ranked number of obligation-level recommenda [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

11 extracted references · 1 canonical work pages · 1 internal anchor

[1]

Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2023. AI-based Question Answering Assistance for Analyzing Natural-language Re- quirements. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1277–1289. doi:10.1109/ ICSE48619.2023.00113

arXiv 2023
[2]

Federico Formica, Nicholas Petrunti, Lucas Bruck, Vera Pantelic, Mark Lawford, and Claudio Menghi. 2023. Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial Simulator. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software ...

doi:10.1145/3611643 2023
[3]

Chao Guo, Tieke He, Wei Yuan, Yue Guo, and Rui Hao. 2020. Crowdsourced requirements generation for automatic testing via knowledge graph. InISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020. ACM, 545–548. doi:10.1145/3395363.3404363

arXiv 2020
[4]

David Boram Hong, Aaron Imani, and Iftekhar Ahmed. 2026. From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills. arXiv:2607.01456 [cs.SE] doi:10.48550/arXiv.2607.01456

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.01456 2026
[5]

Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding Deep Learning System Testing Using Surprise Adequacy. InProceedings of the 41st IEEE/ACM Interna- tional Conference on Software Engineering. IEEE, 1039–1049. doi:10.1109/ICSE. 2019.00108

arXiv 2019
[6]

Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland-Huang. 2021. Traceability Transformed: Generating More Accurate Links with Pre-Trained BERT Models. In43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 324–335. doi:10.1109/ICSE43902. 2021.00040

arXiv 2021
[7]

Zhi Ma, Cheng Wen, Zhexin Su, Xiao Liang, Cong Tian, Shengchao Qin, and Mengfei Yang. 2025. Bridging Natural Language and Formal Specification- Automated Translation of Software Requirements to LTL via Hierarchical Seman- tics Decomposition Using LLMs. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Repu...

arXiv 2025
[8]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test Cov- erage Criteria for RESTful Web APIs. InProceedings of the 10th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation. ACM, 15–21. doi:10.1145/3340433.3342822

arXiv 2019
[9]

Mehdi Mirzaaghaei and Ali Mesbah. 2014. DOM-Based Test Adequacy Criteria for Web Applications. InProceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, 71–81. doi:10.1145/2610384.2610406

arXiv 2014
[10]

Rutherford, Antonio Carzaniga, and Alexander L

Matthew J. Rutherford, Antonio Carzaniga, and Alexander L. Wolf. 2006. Simulation-Based Test Adequacy Criteria for Distributed Systems. InProceedings Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice Conference’17, July 2017, Washington, DC, USA of the 14th ACM SIGSOFT International Symposium on Foundat...

arXiv 2006
[11]

Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy.Comput. Surveys29, 4 (1997), 366–427. doi:10.1145/ 267580.267590

arXiv 1997

[1] [1]

Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2023. AI-based Question Answering Assistance for Analyzing Natural-language Re- quirements. In45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 1277–1289. doi:10.1109/ ICSE48619.2023.00113

arXiv 2023

[2] [2]

Federico Formica, Nicholas Petrunti, Lucas Bruck, Vera Pantelic, Mark Lawford, and Claudio Menghi. 2023. Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial Simulator. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software ...

doi:10.1145/3611643 2023

[3] [3]

Chao Guo, Tieke He, Wei Yuan, Yue Guo, and Rui Hao. 2020. Crowdsourced requirements generation for automatic testing via knowledge graph. InISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020. ACM, 545–548. doi:10.1145/3395363.3404363

arXiv 2020

[4] [4]

David Boram Hong, Aaron Imani, and Iftekhar Ahmed. 2026. From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills. arXiv:2607.01456 [cs.SE] doi:10.48550/arXiv.2607.01456

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.01456 2026

[5] [5]

Jinhan Kim, Robert Feldt, and Shin Yoo. 2019. Guiding Deep Learning System Testing Using Surprise Adequacy. InProceedings of the 41st IEEE/ACM Interna- tional Conference on Software Engineering. IEEE, 1039–1049. doi:10.1109/ICSE. 2019.00108

arXiv 2019

[6] [6]

Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang, and Jane Cleland-Huang. 2021. Traceability Transformed: Generating More Accurate Links with Pre-Trained BERT Models. In43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 324–335. doi:10.1109/ICSE43902. 2021.00040

arXiv 2021

[7] [7]

Zhi Ma, Cheng Wen, Zhexin Su, Xiao Liang, Cong Tian, Shengchao Qin, and Mengfei Yang. 2025. Bridging Natural Language and Formal Specification- Automated Translation of Software Requirements to LTL via Hierarchical Seman- tics Decomposition Using LLMs. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Repu...

arXiv 2025

[8] [8]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test Cov- erage Criteria for RESTful Web APIs. InProceedings of the 10th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation. ACM, 15–21. doi:10.1145/3340433.3342822

arXiv 2019

[9] [9]

Mehdi Mirzaaghaei and Ali Mesbah. 2014. DOM-Based Test Adequacy Criteria for Web Applications. InProceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, 71–81. doi:10.1145/2610384.2610406

arXiv 2014

[10] [10]

Rutherford, Antonio Carzaniga, and Alexander L

Matthew J. Rutherford, Antonio Carzaniga, and Alexander L. Wolf. 2006. Simulation-Based Test Adequacy Criteria for Distributed Systems. InProceedings Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice Conference’17, July 2017, Washington, DC, USA of the 14th ACM SIGSOFT International Symposium on Foundat...

arXiv 2006

[11] [11]

Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy.Comput. Surveys29, 4 (1997), 366–427. doi:10.1145/ 267580.267590

arXiv 1997