Pith. sign in

REVIEW 2 major objections 6 minor 49 references

Assessing Behavioral Validation in UI Component Test Suites Using Inferred Metamorphic Relations

T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that existing UI component test suites exercise substantially more inferred behavioral relations than they explicitly validate, and that a relation-level MR coverage metric exposes this gap where statement and branch covera

desk verdict A well-designed study showing that UI tests exercise far more inferred behavioral relations than they explicitly validate; the Touch/Cover gap is likely real, though the per-type rankings sit on somewhat noisier labels. read the letter →

arxiv 2608.03337 v1 pith:GORWFFVC submitted 2026-08-04 cs.SE

Yu Pei , Cunming Zhang , Jeongju Sohn , Mike Papadakis This is my paper
classification cs.SE
keywords UIcomponenttestingmetamorphicrelationsbehavioralvalidationtestadequacyoracleproblemweak-oracletestsLLM-assistedanalysiscoveragemetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UI component libraries are usually judged by how much of their code tests execute, but executing a behavior and checking that it behaves correctly are different things. This paper makes that difference measurable by turning inferred metamorphic relations—statements of how a component's observable behavior should change or stay invariant under transformations of props, state, interaction, or context—into a behavioral reference space for existing tests. For 214 components from four open-source UI libraries, it aligns each test with the inferred relations and computes two ratios: Touch (behavior exercised) and Cover (behavior explicitly validated). The central result is a large and consistent gap: Cover stays between 42.5% and 47.6% while Touch is far higher, and most uncovered relations are weak-oracle cases where tests run the behavior without checking the expected relation. A sympathetic reader would take this as evidence that execution coverage overstates UI test adequacy, and that a relation-level metric can reveal where the missing validation is.

What carries the argument

The load-bearing object is the inferred metamorphic relation (MR): a structured statement of a source condition, a transformation of inputs, state, interactions, or context, and an expected behavioral relation, grounded in component artifacts. Around it sits a six-category UI-specific taxonomy covering input/prop behavior, state/event semantics, interaction/accessibility, visual/layout behavior, composition/context behavior, and data flow, which constrains LLM inference to consistent relation types. The analytical core is the Touch/Cover distinction: a hybrid deterministic-plus-semantic alignment labels each MR as either touched (exercised) or covered (exercised and explicitly validated), an

What would settle it

Take the 119 failing assertions from the paper's weak-oracle study and adjudicate each against an independent hand-written behavioral specification. If most of the 84 'MR mismatch' failures turn out to be genuine component defects or undocumented behaviors, the weak-oracle gap is real; if most are invalid inferred relations, the gap is an artifact of the reference space. A complementary check: run the framework on a component whose test suite is known to explicitly verify every documented behavior—Cover should approach Touch if the metric measures validation rather than inference noise.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that metamorphic relations (MRs) inferred from a component's public API, documentation, and implementation—treated not as a complete specification but as an empirical behavioral reference—can be aligned to existing tests and converted into two coverage metrics: $Touch$, the proportion of inferred relations exercised by at least one test, and $Cover$, the proportion also explicitly validated by a relation-relevant assertion. Across 214 components from four open-source UI libraries and three LLM configurations, $Cover$ stays between 42.5% and 47.6% while $Touch$ is substantially higher, so the dominant gap is weak-oracle cases: tests run the behavior but

Load-bearing premise

The measurement stands or falls on whether the LLM-inferred MR space is an accurate and complete enough behavioral reference for the studied components, and on whether the hybrid alignment labels—Cover in particular, with 81.6% sampled precision—are correct enough that errors do not systematically distort the reported Touch/Cover gap; the paper itself cautions that its validation is sample-level reliability evidence, not population-level estimates.

Editorial extensions

If this is right

  • Test suites can be profiled as high-Touch/low-Cover or low-Touch, and the two profiles call for different fixes: add relation-relevant assertions versus add test scenarios.
  • MR Cover is a genuine second adequacy axis: within suites of equal size it correlates only weakly with statement and branch coverage, so a component can have high structural coverage and still fail to validate most inferred relations.
  • The weak-oracle gap is concentrated in identifiable relation types—visual/layout, accessibility, and context propagation—giving maintainers a concrete checklist of under-validated behavior.
  • The framework's numbers are stable enough across three LLM configurations that the Touch-over-Cover ordering and the rough size of the gap persist regardless of the model used.
  • Because the framework consumes only source, documentation, and tests, it can be applied to other component libraries without changing the inference or alignment procedure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the consistency of the Touch/Cover gap across three LLMs suggests the weak-oracle pattern is likely a property of current UI component test-writing practice—interaction-heavy and assertion-light—rather than an artifact of one model. A direct test would be to measure whether suites written under a 'relation-relevant assertion required' guideline close the gap.
  • Editorial inference: since the paper's own validation gives Cover a precision of 81.6% on the sampled alignment, aggregate Touch/Cover numbers are safer to use than any individual relation label; a practitioner should treat a single Weak flag as a lead to inspect rather than as a proven missing oracle.
  • Editorial inference: the injected-fault gradient (60% detection for Covered, 47.5% for Weak, 0% for Untouched) suggests that strengthening weak oracles may deliver fault-detection gains comparable to adding new test scenarios; a controlled experiment that adds assertions to half of a random set of Weak relations and new tests to the other half could quantify this directly.
  • Editorial inference: the taxonomy's six categories leave some real-world behaviors unmapped—27.6% of issue annotations have no clear MR match—so the reported coverage percentages should be read as coverage of the inferred behavioral space, not of all possible component behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes a framework that uses inferred metamorphic relations (MRs) as an empirical behavioral reference for assessing UI component test suites. It defines a six-category MR taxonomy, uses LLMs to infer component-specific MRs from source, documentation, and tests, and aligns existing tests to these MRs through a hybrid deterministic-semantic procedure, distinguishing behavioral reach (Touch) from behavioral validation (Cover). On 214 components from four UI libraries, the evaluation reports that Cover (42.5–47.6% across three LLM configurations) is consistently below Touch (roughly 78–89%), that most uncovered relations are weak-oracle cases, and that MR coverage is only weakly correlated with statement/branch coverage. Practical relevance is examined via issue-description mapping, an oracle-strengthening study, and an MR-relevant injected-fault study. The inference and alignment are manually validated on samples.

Significance. If the findings hold, the paper offers a novel and useful relation-level adequacy signal for UI component testing, complementing execution-based coverage and addressing the oracle gap. The framework is clearly specified, the evaluation is extensive (214 components, four libraries, three LLMs, ablations, manual validation), and the Touch/Cover distinction is intuitive and actionable. The paper also credits its limitations (sample-level reliability, inferred rather than complete MR space) and avoids overclaiming causality in the external relevance studies. These strengths make the contribution potentially valuable to the software testing community, provided the reliability concerns around the label validation are properly addressed.

major comments (2)
  1. [IV-C, Table II] The central Touch/Cover results rest on cover labels whose validation is sample-level and whose error distribution is not characterized across relation types. Hybrid Cover precision is 81.6%, i.e., roughly 18% of predicted Cover positives are false positives, and the validation set is stratified by predicted label but not by relation type or library. If these errors concentrate in specific relation types (e.g., visual/layout), the per-type rankings in Figure 4 and the magnitude of the weak-oracle gap in RQ2 could shift. The paper acknowledges this ('sample-level reliability evidence rather than population-level estimates'), but no sensitivity analysis or relation-stratified validation is provided. Please add a bounded analysis (e.g., recompute Touch/Cover under worst-case label error redistribution) or stratify the validation by relation type to show the conclusions are robust to label n
  2. [IV-C, MR validity] The inferred MR space is the denominator for all Touch/Cover computations, yet only 10% (420 MRs) are manually validated, with 88.6% usable, 6.4% invalid/unsupported, and 5.0% duplicates. The framework filters invalid and merges duplicates, but the net effect on the MR count and on the reported coverage percentages is not quantified. A sensitivity check varying the invalid/duplicate rate within the observed range would clarify how much the aggregate numbers could change under alternative filtering decisions.
minor comments (6)
  1. [Throughout] Notation is inconsistent: the text uses MRTouch/MRCover and also 'MR Touch'/'MR Cover' (e.g., Tables III and IV). Please unify, e.g., Touch and Cover, with a single definition for the reader.
  2. [Table I] The column header 'Comp.(LOC)' with subheadings 'Src' and 'Test' is ambiguous. Clarify that Src and Test are average lines of code per component; also indicate whether the reported numbers are averages across components or totals.
  3. [Figure 4] The text says the low-Cover ranking is consistent across LLMs, but Figure 4 appears to show a single ranking. State explicitly which LLM configuration is used in Figure 4 and, if space permits, include the other configurations in an appendix or supplementary material.
  4. [IV-C.b] The absence of an inter-rater agreement statistic is explained by the adjudication procedure, but reporting raw agreement (e.g., percentage of cases where the second reviewer agreed) would help the reader assess annotation reliability.
  5. [V-C, Table IV] The within-size Spearman correlation between statement coverage and MR Touch drops from 0.41 at 25% suite size to -0.12 at full suite. This non-monotonic behavior is unexplained; add a brief interpretation or investigate whether it is an artifact of ceiling effects or the component sample.
  6. [V-D, Table VIII] The injected-fault study reports a 'trend' (60.0% vs 47.5% vs 0%), but no statistical test or confidence interval is provided. A proportion test or a note on sample size would strengthen the claim and help distinguish signal from sampling variability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; central derivation is self-contained, only minor non-load-bearing self-citations.

full rationale

The central claim—that existing UI component test suites exercise more inferred behavioral relations than they explicitly validate, and that MR coverage complements execution-based coverage—is derived from an empirical pipeline. The MR space is inferred from component public APIs, documentation, and implementation (Section III-C), not from the test suite's assertions; Touch and Cover are then computed by aligning existing tests against this independently constructed reference space (Section III-D, Section III-E). The gap between Touch and Cover is a measurement outcome, not a fitted parameter or a quantity defined into existence: the distribution of uncovered relations into Weak versus Untouched is empirical (Figure 3), and the weak correlations with statement/branch coverage (Table IV) are computed from data. The manual validation of MR usability and alignment labels is a reliability check, not a source of the main result. The injected-fault and issue-mapping studies are external supporting analyses, and the paper explicitly labels the zero detection rate for Untouched relations as a sanity-check baseline rather than evidence of oracle strength. The paper does contain self-citations to prior work by the same authors (e.g., [26], [33], [49]), but these are background references for LLM-based MR inference, mutation testing surveys, and GUI test smells; none of them is load-bearing for the paper's central derivation. No equation is shown to reduce to its own input, and no fitted parameter is renamed as a prediction. Overall, the derivation is self-contained and the reported findings do not appear to be forced by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework rests on the reliability of the LLM-inferred MR space and the alignment labels. Both are validated on manually reviewed samples, not exhaustively, so measurement error propagates into the reported Touch/Cover numbers. No numeric free parameters were fitted; the MR space is inferred, not estimated.

assumptions (4)
  • domain assumption The six-category UI MR taxonomy adequately spans the behavioral contracts implied by UI component APIs and documentation.
    Introduced in Section III-B. If the taxonomy misses relation types, the inferred MR space undercounts behaviors and Touch/Cover are biased.
  • domain assumption LLM inference conditioned on component profiles produces MRs that are usable as an empirical behavioral reference.
    Section IV-C reports 88.6% usability on a 420-MR stratified sample; the remaining 11.4% (invalid or duplicate) could bias the MR space.
  • domain assumption Hybrid deterministic and semantic alignment correctly identifies Touch and Cover labels.
    Section IV-C reports sample-level F1 of 96.6% for Touch and 89.7% for Cover, with Cover precision of 81.6%, so false-positive Cover labels remain possible.
  • domain assumption Statement and branch coverage instrumentation via native test infrastructure faithfully measures execution coverage.
    Section IV-D uses native test infrastructure for the 201 components with runtime coverage; this is taken as a standard baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Behavioral Validation in UI Component Test Suites Using Inferred Metamorphic Relations." pith.science (2026). https://pith.science/paper/GORWFFVC

@misc{pith2026260803337,
  author       = {Pith},
  title        = {Pith review of: Assessing Behavioral Validation in UI Component Test Suites Using Inferred Metamorphic Relations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GORWFFVC}},
  note         = {Machine review of arXiv:2608.03337}
}
read the original abstract

UI component libraries are commonly assessed using execution-based metrics such as statement and branch coverage, yet these metrics provide limited insight into whether tests verify the behavioral relations implied by component APIs and documentation. This paper presents an MR-based framework that uses inferred metamorphic relations (MRs) as an empirical behavioral reference, rather than a complete specification, for assessing UI component test suites. Given a component's source, documentation, and tests, the framework infers component-specific MRs using a UI-specific taxonomy, aligns tests with the inferred relations through hybrid deterministic and semantic analysis, and computes relation-level MR coverage metrics. We manually validate both the inferred MR space and the test--MR alignment. Our evaluation shows that existing test suites exercise substantially more behavioral relations than they explicitly validate: MR Cover remains between 42.5% and 47.6% across three LLM configurations and consistently below MR Touch. Most uncovered relations are weak-oracle cases, where behaviors are exercised but lack explicit behavioral validation. MR coverage also complements execution-based coverage by revealing behavioral gaps not reflected by statement or branch coverage alone. We further assess practical relevance through issue-description mapping, oracle strengthening, and MR-relevant injected faults. Most reported issue descriptions can be mapped to inferred MR relation types; weak-oracle relations often expose missing validation evidence; and MR labels show a trend in MR-relevant fault detection. Overall, MR coverage provides a complementary relation-level perspective for assessing behavioral validation in modern UI component testing.

Figures

Figures reproduced from arXiv: 2608.03337 by the authors.

Figure 1
Figure 1. Overview of the MR-based behavioral validation framework. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example: placement consistency MR is inferred from [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Relation types with the lowest Cover rates, grouped by top-level categories. Colored brackets distinguish the groups (blue: Visual/layout, orange: Input/prop, green: Composition/ context, brown: Interaction/accessibility). animation consistency (18.8%), visual alignment (27.9%), and responsive sizing (34.8%). The breakdown further shows that low Cover can arise from different sources. For overflow handling, the gap … view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Distribution of MR coverage outcomes (Cover, Weak, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 5
Figure 5. Figure 5: Relationship between mapped issue annotations and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 40 canonical work pages

  1. [1]

    Reusing ui elements with model-based user interface development,

    A. Delgado, A. Estepa, J. Troyano, and R. Estepa, “Reusing ui elements with model-based user interface development,”International Journal of Human-Computer Studies, vol. 86, pp. 48–62, 2016

    work page 2016

  2. [2]

    Why do developers use trivial packages? an empirical case study on npm,

    R. Abdalkareem, O. Nourry, S. Wehaibi, S. Mujahid, and E. Shihab, “Why do developers use trivial packages? an empirical case study on npm,” inProceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 385–395

    work page 2017

  3. [3]

    Modern front end web archi- tectures with react. js and next. js,

    M. F. S. Lazuardy and D. Anggraini, “Modern front end web archi- tectures with react. js and next. js,”Research Journal of Advanced Engineering and Science, vol. 7, no. 1, pp. 132–141, 2022

    work page 2022

  4. [4]

    An empirical comparison of dependency network evolution in seven software packaging ecosystems,

    A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” Empirical Software Engineering, vol. 24, no. 1, pp. 381–416, 2019

    2019

  5. [5]

    Awesome react. js (unleash the power of modern ui building),

    P. G. Naik and K. Oza, “Awesome react. js (unleash the power of modern ui building),”International Institute of Organized Research (I2OR), 2023

    work page 2023

  6. [6]

    Dependency-induced waste in continuous integration: An empirical study of unused depen- dencies in the npm ecosystem,

    N. R. Weeraddana, M. Alfadel, and S. McIntosh, “Dependency-induced waste in continuous integration: An empirical study of unused depen- dencies in the npm ecosystem,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 2632–2655, 2024

    work page 2024

  7. [7]

    I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages,

    D. Venturini, F. R. Cogo, I. Polato, M. A. Gerosa, and I. S. Wiese, “I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages,”ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 1–26, 2023

    work page 2023

  8. [8]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE transactions on software engineering, vol. 41, no. 5, pp. 507–525, 2014

    work page 2014

Show all 49 references
  1. [9]

    Accessibility issues in android apps: state of affairs, sentiments, and ways forward,

    A. Alshayban, I. Ahmed, and S. Malek, “Accessibility issues in android apps: state of affairs, sentiments, and ways forward,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1323–1334

    2020

  2. [10]

    Declarui: Bridging design and development with automated declarative ui code generation,

    T. Zhou, Y . Zhao, X. Hou, X. Sun, K. Chen, and H. Wang, “Declarui: Bridging design and development with automated declarative ui code generation,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 219–241, 2025

    2025

  3. [11]

    Semantic web accessibility testing via hierarchical visual analysis,

    M. Bajammal and A. Mesbah, “Semantic web accessibility testing via hierarchical visual analysis,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1610– 1621

    2021

  4. [12]

    Coverage is not strongly correlated with test suite effectiveness,

    L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” inProceedings of the 36th international conference on software engineering, 2014, pp. 435–445

    2014

  5. [13]

    Code coverage for suite evaluation by developers,

    R. Gopinath, C. Jensen, and A. Groce, “Code coverage for suite evaluation by developers,” inProceedings of the 36th international conference on software engineering, 2014, pp. 72–82

    2014

  6. [14]

    Code coverage and postrelease defects: A large-scale study on open source projects,

    P. S. Kochhar, D. Lo, J. Lawall, and N. Nagappan, “Code coverage and postrelease defects: A large-scale study on open source projects,”IEEE Transactions on Reliability, vol. 66, no. 4, pp. 1213–1228, 2017

    2017

  7. [15]

    Is mutation an appropriate tool for testing experiments?

    J. H. Andrews, L. C. Briand, and Y . Labiche, “Is mutation an appropriate tool for testing experiments?” inProceedings of the 27th international conference on Software engineering, 2005, pp. 402–411

    2005

  8. [16]

    Assessing oracle quality with checked coverage,

    D. Schuler and A. Zeller, “Assessing oracle quality with checked coverage,” in2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 2011, pp. 90–99

    2011

  9. [17]

    Metamorphic relations for enhancing system understanding and use,

    Z. Q. Zhou, L. Sun, T. Y . Chen, and D. Towey, “Metamorphic relations for enhancing system understanding and use,”IEEE Transactions on Software Engineering, vol. 46, no. 10, pp. 1120–1154, 2018

    2018

  10. [18]

    Automated metamorphic-relation generation with chatgpt: An experience report,

    Y . Zhang, D. Towey, and M. Pike, “Automated metamorphic-relation generation with chatgpt: An experience report,” in2023 IEEE 47th An- nual Computers, Software, and Applications Conference (COMPSAC). IEEE, 2023, pp. 1780–1785

    2023

  11. [19]

    Metamorphic relation generation: State of the art and research directions,

    R. Li, H. Liu, P.-L. Poon, D. Towey, C.-A. Sun, Z. Zheng, Z. Q. Zhou, and T. Y . Chen, “Metamorphic relation generation: State of the art and research directions,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–25, 2025

    2025

  12. [20]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys (CSUR), vol. 51, no. 1, pp. 1–27, 2018

    2018

  13. [21]

    Metamorphic testing of large language models for natural language processing,

    S. Cho, S. Ruberto, and V . Terragni, “Metamorphic testing of large language models for natural language processing,” in2025 IEEE Inter- national Conference on Software Maintenance and Evolution (ICSME). IEEE, 2025, pp. 174–186

    2025

  14. [22]

    Large language models: The next frontier for variable discovery within metamorphic testing?

    C. Tsigkanos, P. Rani, S. M ¨uller, and T. Kehrer, “Large language models: The next frontier for variable discovery within metamorphic testing?” in2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2023, pp. 678–682

    2023

  15. [23]

    Deeptest: Automated testing of deep-neural-network-driven autonomous cars,

    Y . Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural-network-driven autonomous cars,” inProceedings of the 40th international conference on software engineering, 2018, pp. 303– 314

    2018

  16. [24]

    Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,

    M. Zhang, Y . Zhang, L. Zhang, C. Liu, and S. Khurshid, “Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,” inProceedings of the 33rd ACM/IEEE international conference on automated software engineering, 2018, pp. 132–142

    2018

  17. [25]

    An empirical comparison of compiler testing techniques,

    J. Chen, W. Hu, D. Hao, Y . Xiong, H. Zhang, L. Zhang, and B. Xie, “An empirical comparison of compiler testing techniques,” inProceedings of the 38th International Conference on Software Engineering, 2016, pp. 180–190

    2016

  18. [26]

    Abstraction- aware inference of metamorphic relations,

    A. Nolasco, F. Molina, R. Degiovanni, A. Gorla, D. Garbervetsky, M. Papadakis, S. Uchitel, N. Aguirre, and M. F. Frias, “Abstraction- aware inference of metamorphic relations,”Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 450–472, 2024. [Online]. Available: https://doi.org/10.11...

    2024 doi

  19. [27]

    Improving oracle quality by detecting brittle assertions and unused inputs in tests,

    C. Huo and J. Clause, “Improving oracle quality by detecting brittle assertions and unused inputs in tests,” inProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014, pp. 621–631

    2014

  20. [28]

    Property-based testing in practice,

    H. Goldstein, J. W. Cutler, D. Dickstein, B. C. Pierce, and A. Head, “Property-based testing in practice,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  21. [29]

    React apps with server-side rendering: Next. js,

    H. A. Jartarghar, G. R. Salanke, A. K. AR, S. GS, and S. Dalali, “React apps with server-side rendering: Next. js,”Journal of Telecommunica- tion, Electronic and Computer Engineering (JTEC), vol. 14, no. 4, pp. 25–29, 2022

    2022

  22. [30]

    Misty: Ui prototyping through interactive conceptual blending,

    Y . Lu, A. Leung, A. Swearngin, J. Nichols, and T. Barik, “Misty: Ui prototyping through interactive conceptual blending,” inProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 2025, pp. 1–17

    2025

  23. [31]

    Enhancing user engagement through adaptive ui/ux design: A study on personalized mobile app interfaces,

    Y . Liu, H. Tan, G. Cao, and Y . Xu, “Enhancing user engagement through adaptive ui/ux design: A study on personalized mobile app interfaces,” Computer Science & IT Research Journal, vol. 5, no. 8, pp. 1942–1962, 2024

    1942

  24. [32]

    Llamatouch: A faithful and scalable testbed for mobile ui task automation,

    L. Zhang, S. Wang, X. Jia, Z. Zheng, Y . Yan, L. Gao, Y . Li, and M. Xu, “Llamatouch: A faithful and scalable testbed for mobile ui task automation,” inProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024, pp. 1–13

    2024

  25. [33]

    Mutation testing advances: an analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y . Jia, Y . Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” inAdvances in computers. Elsevier, 2019, vol. 112, pp. 275–378

    2019

  26. [34]

    Evaluating the impact of experimental assumptions in automated fault localization,

    E. Soremekun, L. Kirschner, M. B ¨ohme, and M. Papadakis, “Evaluating the impact of experimental assumptions in automated fault localization,” in2023 IEEE/ACM 45th International Conference on Software Engi- neering (ICSE). IEEE, 2023, pp. 159–171

    2023

  27. [35]

    Metamorphic test- ing: a new approach for generating next test cases,

    T. Y . Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic test- ing: a new approach for generating next test cases,”arXiv preprint arXiv:2002.12543, 2020

    2002 arXiv

  28. [36]

    A survey on metamorphic testing,

    S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cort ´es, “A survey on metamorphic testing,”IEEE Transactions on software engineering, vol. 42, no. 9, pp. 805–824, 2016

    2016

  29. [37]

    How effectively does metamorphic testing alleviate the oracle problem?

    H. Liu, F.-C. Kuo, D. Towey, and T. Y . Chen, “How effectively does metamorphic testing alleviate the oracle problem?”IEEE Transactions on Software Engineering, vol. 40, no. 1, pp. 4–22, 2013

    2013

  30. [38]

    Towards generating executable metamorphic relations using large language mod- 11 els,

    S. Y . Shin, F. Pastore, D. Bianculli, and A. Baicoianu, “Towards generating executable metamorphic relations using large language mod- 11 els,” inInternational Conference on the Quality of Information and Communications Technology. Springer, 2024, pp. 126–141

    2024

  31. [39]

    Genmorph: Automatically generating metamorphic relations via genetic programming,

    J. Ayerdi, V . Terragni, G. Jahangirova, A. Arrieta, and P. Tonella, “Genmorph: Automatically generating metamorphic relations via genetic programming,”IEEE Transactions on Software Engineering, vol. 50, no. 7, pp. 1888–1900, 2024

    1900

  32. [40]

    Selecting and constraining metamorphic relations,

    A. Duque-Torres, “Selecting and constraining metamorphic relations,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 212–216

    2024

  33. [41]

    Can llms reason about program semantics? a comprehensive evaluation of llms on formal specification inference,

    T. Le-Cong, B. Le, and T. Murray, “Can llms reason about program semantics? a comprehensive evaluation of llms on formal specification inference,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 21 991–22 014

    2025

  34. [42]

    Specgen: Automated generation of formal program specifications via large language models,

    L. Ma, S. Liu, Y . Li, X. Xie, and L. Bu, “Specgen: Automated generation of formal program specifications via large language models,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 16–28

    2025

  35. [43]

    Hierarchical gui test case generation using automated planning,

    A. M. Memon, M. E. Pollack, and M. L. Soffa, “Hierarchical gui test case generation using automated planning,”IEEE transactions on software engineering, vol. 27, no. 2, pp. 144–155, 2001

    2001

  36. [44]

    Designing and comparing automated test oracles for gui-based software applications,

    Q. Xie and A. M. Memon, “Designing and comparing automated test oracles for gui-based software applications,”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 16, no. 1, pp. 4–es, 2007

    2007

  37. [45]

    Combodroid: generating high-quality test inputs for android apps via use case combinations,

    J. Wang, Y . Jiang, C. Xu, C. Cao, X. Ma, and J. Lu, “Combodroid: generating high-quality test inputs for android apps via use case combinations,” inProceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 469–480

    2020

  38. [46]

    Vet: identifying and avoiding ui exploration tarpits,

    W. Wang, W. Yang, T. Xu, and T. Xie, “Vet: identifying and avoiding ui exploration tarpits,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 83–94

    2021

  39. [47]

    Testing web enabled simulation at scale using metamorphic testing,

    J. Ahlgren, M. Berezin, K. Bojarczuk, E. Dulskyte, I. Dvortsova, J. George, N. Gucevska, M. Harman, M. Lomeli, E. Meijeret al., “Testing web enabled simulation at scale using metamorphic testing,” in 2021 IEEE/ACM 43rd International Conference on Software Engineer- ing: Softwa...

    2021

  40. [48]

    On the relation of test smells to software code quality,

    D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli, “On the relation of test smells to software code quality,” in2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE Computer Society, 2018, pp. 1–12

    2018

  41. [49]

    Smells in system user interactive tests,

    R. Rwemalika, S. Habchi, M. Papadakis, Y . Le Traon, and M.-C. Brasseur, “Smells in system user interactive tests,”Empirical Software Engineering, vol. 28, no. 1, p. 20, 2023. 12

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.