Pith. sign in

REVIEW 4 major objections 4 minor 62 references

A pipeline that feeds LLMs hierarchically organized repository context and then runs a debate-based refinement stage can generate thousands of logically valid, directly testable metamorphic relations for augmented-reality applications, the

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:23 UTC pith:2BEPPRMP

load-bearing objection A useful ablation of context granularity for LLM-generated metamorphic relations, but the mutation-based reliability claim is circular and needs to be redone. the 4 major comments →

arxiv 2607.28775 v1 pith:2BEPPRMP submitted 2026-07-30 cs.SE

Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models

Dibyendu Brinto Bose , Jiawei Qin , Chris Brown This is my paper
classification cs.SE
keywords metamorphic testingmetamorphic relation generationaugmented realitylarge language modelsrepository-level contexttest oracle generationLLM debate refinementproperty-based testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that LLMs can generate metamorphic relations—expected input–output relationships used in testing—for AR applications at scale, even though defining exact test oracles for AR is notoriously difficult. It argues that how code context is organized matters as much as how much context is given: a hierarchical structure (repository summary, class summary, method signature) outperforms both method-only and flat-context prompts in coverage, diversity, and traceability. It also argues that a debate-style refinement stage, where a judging model detects contradictions and redundancies among candidates and selects, merges, or discards them, is necessary to turn raw candidates into coherent, testable relations. If correct, developers of AR systems could obtain test oracles automatically instead of hand-crafting them.

Core claim

On its own terms, the paper introduces a pipeline with three context configurations—A0 (method-only), A1 (flat retrieved repository context), and A2 (hierarchical repository context)—followed by a reasoning-based deliberation stage. It reports that the hierarchical configuration A2 generates the broadest coverage (7,004 metamorphic relations across 142 repositories and 5,167 class–method pairs) with the lowest exact duplicate rate (13.3%), that deliberation detects contradictions in 79.0% of cases, reduces the exact duplicate rate to 1.3%, and selects A2-derived relations in 88.2% of outcomes. A manual oracle study of 141 refined relations finds mean validity 1.80/2 and testability 1.82/2, a

What carries the argument

Two mechanisms carry the argument. First is the hierarchical context representation (A2): three nested prompt layers—a repository-level summary, a class-level summary with inheritance and dependency relations, and the target method signature—that preserve the object-oriented structure of Unity AR codebases and supply behavioral signal that flat context blurs. Second is the critique-based deliberation stage: a judging model receives all candidate MRs for a target method, identifies contradictions and redundancies, and emits one of three decisions—KEEP_BEST, MERGE, or DISCARD_ALL. The hierarchy provides architectural intent and localized implementation detail; the deliberation reconciles compe

Load-bearing premise

The load-bearing premise is that the deliberation judge's decisions genuinely improve MR quality rather than merely preferring the hierarchical-context candidates; the paper's manual validation found only 31–37% of MERGE decisions clearly correct (Section 5.2), and the authors acknowledge a possible judge bias toward A2 (Section 6.2).

What would settle it

Replace the Qwen2.5-Coder-32B judge with a judge matched in size to the 3B generator (or vice versa) and re-run the debate; if A2's 88.2% win rate does not persist, the hierarchy preference is an artifact of judge bias. Alternatively, evaluate raw A2 candidates against debate-refined MRs under the same human rubric; if raw A2 matches or beats the refined set on validity and testability, the refinement stage adds no value.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AR testing no longer requires hand-written oracles: generated metamorphic relations can be translated directly into property-based test assertions.
  • Structured context organization becomes a first-class design decision for LLM-based code tasks; method-only or flat prompts will underperform in coverage and source traceability.
  • The deliberation stage reduces redundancy sharply (exact duplicates from 13.3% to 1.3%) and produces merged relations that rate higher on validity, AR specificity, and testability than single-best selections.
  • Repository-scale MR generation is feasible with a small generator: 7,004 relations across 5,167 class–method pairs in 142 repositories using a 3B-parameter model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: If the pipeline generalizes beyond Unity/C# to other oracle-poor domains such as robotics or simulation, the hierarchical-context-plus-debate recipe could become a general template for automatic oracle generation.
  • Editorial extension: A decisive test would be to match the judge model's capacity to the generator (both 3B or both 32B) and re-run the debate; if the 88.2% A2 preference does not persist, the reported hierarchy benefit is partly an artifact of judge bias rather than a property of the context structure.
  • Editorial extension: The mutation-study authors treat mutants that preserve the tested MR property as 'equivalent'; this suggests a formal mutation-equivalence criterion for MR-based testing, where a mutant is equivalent if and only if it satisfies every relation in the MR set.
  • Editorial extension: Because the paper's own manual validation found only 31–37% of MERGE decisions clearly correct, an adaptive refinement loop could fall back to KEEP_BEST when humans flag a merge as dubious, improving trust in the pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a pipeline for generating metamorphic relations (MRs) for AR applications using LLMs. Three context configurations (A0: method-only, A1: flat repository context, A2: hierarchical repository context) are used to generate 14,916 candidate MRs from 142 Unity-based AR repositories. A Qwen2.5-Coder-32B judge then applies debate-based refinement to aggregate candidate sets, choosing KEEP_BEST, MERGE, or DISCARD_ALL. The authors evaluate structural validity, coverage, redundancy, and human-rated validity/testability/AR-specificity, and report a preliminary mutation study on five MRs. They conclude that hierarchical context and deliberation yield scalable, reliable, domain-relevant test oracles.

Significance. If the claims hold, the work would be a useful contribution to automated test oracle generation: it addresses MR discovery at repository scale, provides a large public dataset/artifact, and combines context curation with LLM-based reconciliation. The strongest parts are the deterministic coverage and duplicate-rate measurements, the large scale (142 repositories, 14,916 MRs), the manual oracle study with three raters, and the public artifacts. However, the central 'reliable oracle' claim rests on a mutation study whose 100% detection figure is circular, and the non-triviality metrics in RQ1 are weakened by thresholds derived from the very data they evaluate. These issues are fixable but currently limit the strength of the conclusions.

major comments (4)
  1. [Section 4.3, Eq. (1) and Tables 3-4] The p10 non-triviality thresholds are derived from the pooled set of 14,916 generated MRs and then used to evaluate those same MRs via criteria C2/C3 and the composite SVS. Because p10 is the 10th percentile of the pooled distribution, the overall non-triviality rate is forced to be approximately 90% by construction. This makes the 'non-triviality' metric a relative, data-dependent quantity rather than an absolute quality measure, and it undermines the RQ1 claim that A2 improves specificity. Please use thresholds established on a held-out set or fixed independently justified criteria, and report differences with confidence intervals.
  2. [Section 6.3, 'Results' and Table 9] The mutation study reclassifies all 14 un-killed mutants as 'equivalent with respect to the tested metamorphic property' and then reports 'all non-equivalent mutants (16/16, 100%) were detected.' This is circular: a mutant is classified as non-equivalent exactly when it violates the MR, so the MR cannot miss a non-equivalent mutant by definition. Standard mutation testing defines equivalence at the program level, not with respect to the property under test. The 14 mutants (e.g., changed offsets, centering constants) may well be non-equivalent programs that merely preserve the relational property. The claim of 100% detection is therefore not evidence of oracle reliability. Please re-analyze the 14 mutants independently, or report the unadjusted killed rate (16/30) and discuss what this means for the 'reliable oracle' conclusion.
  3. [Section 5.2, 'Manual Validation of Judge Decisions'] The key RQ2 quantities—88.2% A2 preference and duplicate-rate reduction from 13.3% to 1.3%—are produced by the same judge whose decisions are being validated. The manual validation found that MERGE decisions were only 31–37% clearly correct, and the authors acknowledge in Section 6.2 that the judge's A2 preference may reflect shared biases with hierarchical prompting. Since MERGE is central to the claim that reasoning-based refinement improves quality, this is not sufficient validation. Please provide an independent, outcome-level human evaluation of final refined MRs (not just judge decisions), or at least re-run deliberation with matched-capability models and compare against a keep-best baseline.
  4. [Section 4.5 and Section 5.3] The oracle study evaluates only 141 MRs manually selected by the first author from 3,760 debate-refined MRs, explicitly filtering for AR-specific behaviors. This selection bias means the reported high validity (1.80/2) and testability (1.82/2) rates apply to a curated, non-random subset and cannot be generalized to the full pipeline output. The RQ3 Summary states that 'most generated relations are sufficiently valid and well-specified,' but the evidence only supports that claim for the curated subset. Please report results on a random sample of the full refined set, or clearly limit the claim to the curated subset.
minor comments (4)
  1. [Section 4.1] The dataset is said to come from prior work [8], which appears to be the authors' own ASEW 2025 paper. Please clarify the relationship to that prior dataset and what incremental filtering/processing was done here, so readers can assess novelty and data provenance.
  2. [Table 7] Comparing the 13.3% duplicate rate for A2 with the 1.3% rate for debate-refined outputs mixes different units: the debate-refined set has a different total N (3,760 vs. 7,004) and includes MERGE/DISCARD_ALL outcomes. Clarify whether the duplicate-rate reduction is computed on the same target-method population.
  3. [Section 5.3, Inter-Rater Reliability] The pairwise weighted kappa between Rater 1 and Rater 3 for testability is −0.01, yet the paper summarizes inter-rater reliability as 'slight-to-fair' using Fleiss κ. This hides a systematic disagreement that is relevant to the testability scores. Report per-rater agreement and whether the reported mean scores are based on majority votes or individual ratings.
  4. [General] Minor writing issues: Section 5.1 contains an uncapitalized 'Though' after a comma; Table 4 uses '%≥p10 Threshold' without a space; and Figure 3 is referenced before its caption is described in the text. These are cosmetic but should be fixed.

Circularity Check

1 steps flagged

Mutation case study's 100% non-equivalent detection is tautological: survivors are reclassified as equivalent to the tested MR.

specific steps
  1. self definitional [Section 6.3, 'Execution-Based Validation', Results paragraph (Table 9 discussion)]
    "The generated property-based tests killed 16 out of 30 mutants (53%). The remaining 14 mutants did not violate the corresponding MR properties because they preserved the relational behavior being tested. ... These cases were therefore treated as equivalent with respect to the tested metamorphic property. After excluding such equivalent mutants, all non-equivalent mutants (16/16, 100%) were detected."

    Standard mutation testing defines an equivalent mutant relative to the whole program's input/output behavior, not relative to the relation under test. The paper redefines every surviving mutant as 'equivalent with respect to the tested metamorphic property,' then excludes those survivors from the denominator and reports 100% detection of 'non-equivalent' mutants. Since the generated tests are literal implementations of the MR, any mutant that actually violates the MR will fail by construction. The term 'non-equivalent' has thus been redefined to mean 'violates (and is detected by) the MR,' making the 100% figure an artifact of the classification rather than evidence of oracle strength. The raw kill rate (16/30, 53%) is the only non-tautological data point, and the excluded examples (change

full rationale

The central 'reliable oracle' claim relies partly on human ratings and manual translation, which are independent, and partly on the execution-based mutation study. That study contains a load-bearing circular step: it classifies the 14 un-killed mutants as equivalent to the tested metamorphic property, so the reported 100% detection of non-equivalent mutants follows by definition. This is not merely a threat; it is a redefinition that turns detection into a tautology. The p10 thresholds are data-derived but used as a fixed normalization and do not force the reported ranking. Self-citations ([8] for the dataset, [9] for AR MRs) provide inputs and motivation but are not the load-bearing derivation. The judge-preference concern is a confound, not a circularity. Because one key piece of evidence reduces by construction while substantial independent human evaluation remains, the score is 6.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The pipeline rests on a handful of author-chosen thresholds and domain assumptions; no new physical entities. The most consequential free parameter is the p10 threshold set, since it defines 'non-triviality' in the main RQ1 result.

free parameters (3)
  • p10 non-triviality thresholds = input_transformation >= 30 chars; output_relation >= 34 chars
    Computed from the pooled set of 14,916 generated MRs (Section 4.3); used as data-derived thresholds to define non-triviality, inflating specificity pass rates relative to an external standard.
  • High-quality threshold for oracle study = Not precisely stated; 'majority high' vs 'consensus high' in Table 8; 61.7% high-quality
    The threshold for 'high-quality' across validity/AR specificity/testability is author-defined and not externally anchored.
  • Temperature and max tokens = temperature=0.3, max_new_tokens=1024
    Chosen by hand for generation and deliberation; not swept, so stochastic effects unquantified.
axioms (4)
  • domain assumption Dataset of 142 Unity C# AR repositories is representative and correctly extracted
    Dataset comes from prior work by the same authors [8]; no external validation of representativeness (Section 4.1).
  • domain assumption LLM-generated candidate MRs are meaningful behavioral hypotheses
    No ground truth for MRs exists; the pipeline assumes LLM outputs are a reasonable candidate space (Section 3.2).
  • domain assumption Debate judge reliably identifies contradictions and redundancies
    Manual validation of 500 decisions shows high agreement for KEEP_BEST/DISCARD_ALL but only 31-37% clearly correct for MERGE (Section 5.2).
  • domain assumption Mutation operators produce realistic faults
    AOR/UOI/SVR/SDL applied mechanically; no guarantee they model real AR bugs (Section 6.3).

pith-pipeline@v1.3.0-alltime-deepseek · 18635 in / 11041 out tokens · 106392 ms · 2026-08-03T00:23:40.826086+00:00 · methodology

0 comments
read the original abstract

Metamorphic Testing (MT) provides a promising approach for testing software without defined test oracles by specifying expected relations between inputs and outputs, instead of relying on exact outputs. For example, testing Augmented Reality (AR) applications is challenging due to dynamic interactions between virtual content, physical environments, and code, which make traditional test oracles difficult to define. However, formulating metamorphic relations (MRs) is time-consuming and burdensome. We introduce a context-aware pipeline that generates and refines MRs using repository-level context and reasoning orchestration, evaluated on a dataset of 142 mobile AR system repositories. Across three context configurations generating 14,916 candidate MRs, hierarchical context yielded the broadest coverage (7,004 MRs across 142 repositories and 5,167 class--method pairs) and lower redundancy. An agentic deliberation process then reconciled conflicting candidates---observed in 79.0% of cases---reducing duplication and selecting context-aware relations in 88.2% of outcomes. A manual oracle study shows refined relations (n = 141) are both logically valid and sufficiently concrete to be directly translated into test assertions, and a preliminary case study reveals converting generated MRs (n = 5) into executable tests can detect non-equivalent mutations in real-world code. Overall, our results show that combining repository-aware MR generation with reasoning-based refinement enables scalable construction of reliable, domain-relevant test oracles.

Figures

Figures reproduced from arXiv: 2607.28775 by Chris Brown, Dibyendu Brinto Bose, Jiawei Qin.

Figure 1
Figure 1. Figure 1: MR generation pipeline. For each target repository, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Prompt structure used for candidate MR generation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Debate-based MR refinement workflow. Candidate [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example of translating a generated MR into a [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 4 canonical work pages

  1. [1]

    John Ahlgren, Maria Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Erik Meijer, et al. 2021. Testing web enabled simulation at scale using metamorphic testing. In2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 140–149

    2021

  2. [2]

    Emran Altamimi, Abdullah Elkawakjy, and Cagatay Catal. 2023. Metamorphic relation automation: Rationale, challenges, and solution directions.Journal of Software: Evolution and Process35, 1 (2023), e2509

    2023

  3. [3]

    Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Par- mit K. Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY...

    arXiv 2020

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  5. [5]

    Jinsheng Ba, Yuancheng Jiang, and Manuel Rigger. 2025. Metamorphic Coverage. arXiv preprint arXiv:2508.16307(2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chun- yang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzger- ald, et al. 2025. Guidelines for empirical studies in software engineering involving large language models.arXiv preprint arXiv:2508.15503(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey.IEEE transactions on software engineering41, 5 (2014), 507–525

    2014

  8. [8]

    Dibyendu Brinto Bose, Yoseph Berhanu Alebachew, and Chris Brown. 2025. LLMs in Debate: Does Arguing Make Them Better at Detecting Metamorphic Relations?. In2025 40th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). 43–50. doi:10.1109/ASEW67777.2025.00019

    arXiv 2025

  9. [9]

    Dibyendu Brinto Bose, Brendan David-John, and Chris Brown. 2024. Optimizing ar application testing: Integrating metamorphic testing to address developer and end-user challenges. InInternational Conference on Human-Computer Interaction. Springer, 20–33

    2024

  10. [10]

    Arda Celik and Qusay H Mahmoud. 2025. A Review of Large Language Models for Automated Test Case Generation.Machine Learning and Knowledge Extraction 7, 3 (2025), 97

    2025

  11. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  12. [12]

    Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities.ACM Computing Surveys (CSUR)51, 1 (2018), 1–27

    2018

  13. [13]

    Stevão Alves de Andrade, Fatima LS Nunes, and Márcio Eduardo Delamaro. 2023. Exploiting deep reinforcement learning and metamorphic testing to automatically test virtual reality applications.Software Testing, Verification and Reliability33, 8 (2023), e1863

    2023

  14. [14]

    M Eswaran and M V A Raju Bahubalendruni. 2022. Challenges and opportunities on AR/VR technologies for manufacturing systems in the context of industry 4.0: A state of the art review.Journal of Manufacturing Systems65 (2022), 260–278. doi:10.1016/j.jmsy.2022.09.016

    work page doi:10.1016/j.jmsy.2022.09.016 2022

  15. [15]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53

    2023

  16. [16]

    Maribeth Gandy and Blair MacIntyre. 2014. Designer’s Augmented Reality Toolkit, Ten Years Later: Implications for New Media Authoring Tools. InProceed- ings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA)(UIST ’14). Association for Computing Machinery, New York, NY, USA, 627–636. doi:10.1145/2642918.2647369

    arXiv 2014

  17. [17]

    Faraz Gurramkonda, Avishak Chakroborty, Bruce Maxim, Mohamed Wiem Mkaouer, and Foyzul Hassan. 2025. VRTestSniffer: Test Smell Detector for Virtual Reality (VR) Software Projects. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1956–1968. 1https://github.com/brintodibyendu/MR_DISCOVERY_ASE

    2025

  18. [18]

    Lai Yip Hang, Rohaida Romli, and Nadia Diyana Mohd Muhaiyuddin. 2023. AR- OOP: Augmented Reality-based Mobile Learning Application for Learning Object- Oriented Programming.Multidisciplinary Applied Research and Innovation4, 1 (2023), 116–121

    2023

  19. [19]

    Anh Nguyen Hoang, Minh Le-Anh, Bach Le, and Nghi DQ Bui. 2025. CodeWiki: Evaluating AI’s Ability to Generate Holistic Documentation for Large-Scale Codebases.arXiv preprint arXiv:2510.24428(2025)

    Pith/arXiv arXiv 2025

  20. [20]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Sangwon Hyun, Mingyu Guo, and M Ali Babar. 2024. Metal: Metamorphic testing framework for analyzing large-language model qualities. In2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 117–128

    2024

  22. [22]

    Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2025. Codesim: Multi-agent code generation and problem solving through simulation-driven planning and debugging. InFindings of the Association for Computational Lin- guistics: NAACL 2025. 5113–5139

    2025

  23. [23]

    Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007. Deckard: Scalable and accurate tree-based detection of code clones. In29th International Conference on Software Engineering (ICSE’07). IEEE, 96–105

    2007

  24. [24]

    René Just and Franz Schweiggert. 2009. Evaluating testing strategies for imaging software by means of mutation analysis. InSoftware Testing Verification and Validation Workshop, IEEE International Conference on. IEEE Computer Society, 205–209

    2009

  25. [25]

    Upulee Kanewala and James M. Bieman. 2013. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). 1–10. doi:10.1109/ISSRE.2013.6698899

    arXiv 2013

  26. [26]

    Veronika Krauß, Alexander Boden, Leif Oppermann, and René Reiners. 2021. Current Practices, Challenges, and Design Implications for Collaborative AR/VR Application Development. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems(Yokohama, Japan)(CHI ’21). Association for Com- puting Machinery, New York, NY, USA, Article 454, 15 ...

    doi:10.1145/3411764 2021

  27. [27]

    Thomas Laurent, Paolo Arcaini, Xiao-Yi Zhang, and Fuyuki Ishikawa. 2024. Meta- morphic testing of an autonomous delivery robots scheduler. In2024 IEEE Con- ference on Software Testing, Verification and Validation (ICST). IEEE, 361–372

    2024

  28. [28]

    2017.3D user interfaces: theory and practice

    Joseph J LaViola Jr, Ernst Kruijff, Ryan P McMahan, Doug Bowman, and Ivan P Poupyrev. 2017.3D user interfaces: theory and practice. Addison-Wesley Profes- sional

    2017

  29. [29]

    Haiyang Li. 2025. Mrg-bench: Evaluating and exploring the requirements of context for repository-level code generation.arXiv preprint arXiv:2508.02998 (2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. 2025. Swe-debate: Competitive multi-agent debate for software issue resolution.arXiv preprint arXiv:2507.23348(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Nan Li and Jeff Offutt. 2016. Test oracle strategies for model-based testing.IEEE Transactions on Software Engineering43, 4 (2016), 372–395

    2016

  32. [32]

    Rui Li, Huai Liu, Pak-Lok Poon, Dave Towey, Chang-Ai Sun, Zheng Zheng, Zhi Quan Zhou, and Tsong Yueh Chen. 2024. Metamorphic relation generation: State of the art and visions for future research.arXiv preprint arXiv:2406.05397 (2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Rui Li, Huai Liu, Pak-Lok Poon, Dave Towey, Chang-Ai Sun, Zheng Zheng, Zhi Quan Zhou, and Tsong Yueh Chen. 2025. Metamorphic relation generation: State of the art and research directions.ACM Transactions on Software Engineering and Methodology34, 5 (2025), 1–25

    2025

  34. [34]

    Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Qizhe Shieh, and Wenmeng Zhou. 2025. Codexgraph: Bridging large language models and code repositories via code graph databases. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2025

  35. [35]

    Yanwen Liu, Ruifeng Li, Hao Tao, and Zheng Zheng. 2024. Test Adequacy Criteria for Metamorphic Testing. In2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE, 527–534

    2024

  36. [36]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Ad- vancing code review automation with large language models through parameter- efficient fine-tuning. In2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 647–658

    2023

  37. [37]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Design choices made by LLM-based test generators prevent them from finding bugs.arXiv preprint arXiv:2412.14137(2024)

    Pith/arXiv arXiv 2024

  38. [38]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-driven development and llm-based code generation. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1583–1594

    2024

  39. [39]

    Johannes Mayer and Ralph Guderlei. 2006. An Empirical Study on the Selection of Good Metamorphic Relations. In30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 1. 475–484. doi:10.1109/COMPSAC. Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using LLMs Conference acronym ’XX, June...

    work page doi:10.1109/compsac 2006

  40. [40]

    Christian Murphy, Gail Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing. 867–872

    2008

  41. [41]

    Christian Murphy, Mohammad S Raunak, Andrew King, Sanjian Chen, Christo- pher Imbriano, Gail Kaiser, Insup Lee, Oleg Sokolsky, Lori Clarke, and Leon Osterweil. 2011. On effective testing of health care simulation software. In Proceedings of the 3rd workshop on software engineering in health care. 40–47

    2011

  42. [42]

    Islam Nasr, Lobna Nassar, and Fakhri Karray. 2024. A study of the interactive role of metamorphic testing and machine learning in the quality assurance of a deep learning forecasting application.International Journal of Information Technology 16, 1 (2024), 105–120

    2024

  43. [43]

    Amirkia Rafiei Oskooei, Selcan Yukcu, Mehmet Cevheri Bozoglan, and Mehmet S Aktas. 2025. Repository-level code understanding by llms via hierarchical summa- rization: Improving code search and bug localization. InInternational Conference on Computational Science and Its Applications. Springer, 88–105

    2025

  44. [44]

    Hellendoorn

    Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn

  45. [45]

    Faqeer Ur Rehman and Madhusudan Srinivasan. 2023. Metamorphic testing for machine learning: Applicability, challenges, and research opportunities. In 2023 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE, 34–39

    2023

  46. [46]

    José Miguel Rojas, Mattia Vivanti, Andrea Arcuri, and Gordon Fraser. 2017. A detailed investigation of the effectiveness of whole test suite generation.Empirical Software Engineering22, 2 (2017), 852–893

    2017

  47. [47]

    Kira Sam. 2024. Llama 3.1: An in-depth analysis of the next-generation large language model.A vailable at SSRN 6139407(2024)

    2024

  48. [48]

    Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing.IEEE Transactions on software engineering42, 9 (2016), 805–824

    2016

  49. [49]

    Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Meta- morphic testing of RESTful web APIs. InProceedings of the 40th International Conference on Software Engineering. 882–882

    2018

  50. [50]

    Lijun Shan and Hong Zhu. 2009. Generating structurally complex test cases by data mutation: A case study of testing an automated modelling tool.Comput. J. 52, 5 (2009), 571–588

    2009

  51. [51]

    Hall, Ao Yu, Bowen Zhang, Haihua Zhang, Janet Nebeling, and Michael Nebeling

    Maximilian Speicher, Brian D. Hall, Ao Yu, Bowen Zhang, Haihua Zhang, Janet Nebeling, and Michael Nebeling. 2018. XD-AR: Challenges and Opportunities in Cross-Device Augmented Reality Application Development.Proc. ACM Hum.- Comput. Interact.2, EICS, Article 7 (jun 2018), 24 pages. doi:10.1145/3229089

    work page doi:10.1145/3229089 2018

  52. [52]

    Chang-ai Sun, Hui Jin, SiYi Wu, An Fu, ZuoYi Wang, and Wing Kwong Chan. 2024. Identifying metamorphic relations: A data mutation di- rected approach.Software: Practice and Experience54, 3 (2024), 394–418. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.3280 doi:10.1002/spe. 3280

    work page doi:10.1002/spe.3280 2024

  53. [53]

    Christoph Treude. 2024. Qualitative data analysis in software engineering: Tech- niques and teaching insights. InHandbook on Teaching Empirical Software Engi- neering. Springer, 155–176

    2024

  54. [54]

    Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using pre-trained models to boost code review automation. InProceedings of the 44th international conference on software engineering. 2291–2302

    2022

  55. [55]

    Dongwei Xiao, Zhibo Liu, Yuanyuan Yuan, Qi Pang, and Shuai Wang. 2022. Metamorphic testing of deep learning compilers.Proceedings of the ACM on Measurement and Analysis of Computing Systems6, 1 (2022), 1–28

    2022

  56. [56]

    Congying Xu, Songqiang Chen, Jiarong Wu, Shing-Chi Cheung, Valerio Terragni, Hengcheng Zhu, and Jialun Cao. 2024. Mr-adopt: Automatic deduction of input transformation function for metamorphic testing. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 557–569

    2024

  57. [57]

    Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024. Execrepobench: Multi-level executable code completion evaluation.arXiv preprint arXiv:2412.11990(2024)

    Pith/arXiv arXiv 2024

  58. [58]

    Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, and Hong Mei. 2014. Search-based inference of polynomial metamorphic relations. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 701–712

    2014

  59. [59]

    Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2015. Metamorphic test- ing for software quality assessment: A study of search engines.IEEE Transactions on Software Engineering42, 3 (2015), 264–284

    2015

  60. [60]

    Zhengyang Zhu, Hong-Ning Dai, Hanyang Guo, Zeqin Liao, and Zibin Zheng

  61. [2023]

    In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    CAT-LM Training Language Models on Aligned Code And Tests. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 409–420. doi:10.1109/ASE56229.2023.00193 ISSN: 2643-1572

    arXiv 2023

  62. [2025]

    In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    VRExplorer: A Model-based Approach for Semi-Automated Testing of Vir- tual Reality Scenes. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 482–494