Pith. sign in

REVIEW 2 major objections 5 minor 57 references

Search-Based Generation of Undetected Quantum Circuit Mutants

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces QUMUG, a search-based generator of quantum circuit mutants tuned to pass a given test suite while remaining non-equivalent, and reports that these mutants are three times harder to detect than those from existing tools.

desk verdict Novel approach, broken validation: the non-equivalence test suffers from massive multiple testing, inflating the headline results. read the letter →

arxiv 2608.09318 v1 pith:55LUW5CR submitted 2026-08-10 cs.SE

Eñaut Mendiluze Usandizaga , Thomas Laurent , Paolo Arcaini , Shaukat Ali This is my paper · ORCID
classification cs.SE
keywords quantummutationtestingcircuitmutantssearch-basedsoftwareengineeringgeneticalgorithmparameterisablegateshigher-ordertestsuiteimprovement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces QUMUG, a search-based generator of quantum circuit mutants that are tuned to pass a given test suite while remaining behaviourally different from the original circuit. Existing quantum mutation tools apply fixed-gate changes that are easy for almost any test to catch, so QUMUG replaces them with parameterised rotation and universal gates and lets a search algorithm tune the angles. On 30 circuits from six quantum algorithms, the authors report that the genetic algorithm found at least one non-equivalent, undetected mutant in 99.67% of runs, generated an average of 494 such mutants per program with a 94.3% non-equivalence ratio, and produced mutants that are detected by about three times fewer test cases and require a median of five new test cases to catch, versus one for the baselines. The paper also presents the first higher-order quantum circuit mutants and shows that combining two mutations often changes which test cases detect the fault, in ways not predictable from the individual mutations.

What carries the argument

The load-bearing mechanism is the parameterised-gate mutation operator combined with a two-part fitness function. Each candidate mutant encodes a circuit position, one of five operators (add $R_x$, add $R_y$, add $R_z$, add $U$, replace with $U$), and three floating-point angles restricted in practice to the range $[-0.5,0.5]$. The fitness function minimises the number of test cases that detect the mutant, rewarding closeness to the undetected region; once no test case detects it, the function maximises total Hellinger distance between the original and mutated output distributions, so the search does not simply converge on equivalent mutants. Detection in the evaluation is decided by a chi-square test, and a five-times-larger comprehensive suite is used only after generation to classify mutants as non-equivalent.

What would settle it

Apply an exact state-vector equivalence check to the mutants that QUMUG classifies as non-equivalent; if a substantial fraction of those mutants are exactly equivalent under full-state comparison, then the reported non-equivalence ratio and test-suite-improvement numbers overstate the method's value.

Watch

Extended reading notes

Core claim

The central claim is that mutant difficulty is controllable by making the mutation itself parameterised. Instead of inserting a fixed gate, QUMUG adds or replaces gates drawn from $R_x(\theta)$, $R_y(\theta)$, $R_z(\phi)$, and the universal $U(\theta,\phi,\lambda)$ gate, and the search tunes the angles so the mutant passes every test in the input suite while still producing a measurable difference in output distributions. The paper's evaluation concludes that this approach produces mutants that are markedly more challenging than fixed-gate mutants: the best search algorithm, a genetic algorithm, required a median of five additional test cases to reveal all of its mutants, whereas mutants from the baseline tools were all detected after adding a single test case. The higher-order analysis further shows that second-order mutants, meaning two mutations in the same circuit and most often placed on different qubits or circuit positions, are detected by a set of test cases different from the union of the detection sets of their constituent first-order mutants in 71.25% of cases, evidence that interactions between faults create behaviours not present in either mutation alone.

Load-bearing premise

The result rests on the assumption that a mutant classified as non-equivalent because the expanded test suite detects it really does differ from the original circuit, since that expanded suite is still a finite sample of initialisations and measurement bases.

Editorial extensions

If this is right

  • Quantum mutation scores computed with fixed-gate tools likely overestimate test-suite quality, because the easy-to-detect mutants inflate kill rates; QUMUG's surviving mutants give a stricter benchmark.
  • Surviving mutants can be used directly as test-generation targets: each new mutant that passes the suite pinpoints a missing test case, so suites can be strengthened iteratively until the search stalls.
  • Higher-order quantum mutants should be treated as interaction faults rather than paired first-order faults, since their detection sets usually differ from the union of their components.
  • The parameterised-gate operators can be reused outside mutation analysis, for instance in fault localisation and circuit repair, where controlling the size of a change is useful.
  • Diversity, not just difficulty, matters for test-suite improvement: the paper attributes the genetic algorithm's advantage partly to its maintaining a more diverse mutant population.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported difficulty gap depends on the chosen oracle: a different statistical test, distance metric, or a state-vector comparator could plausibly shrink or widen the three-fold gap between QUMUG and the fixed-gate baselines.
  • Because the search already treats each mutant as an array of positions and angles, QUMUG could be embedded in a continuous-integration loop that adds every generated survivor to the test suite and reruns the search until no new non-equivalent mutants appear.
  • The dominance of second-order mutants placed on different qubits or positions suggests a testable interaction hypothesis: separation between the two mutations may control whether effects cancel or compound, and sweeping that separation directly would reveal the mechanism.
  • The restriction of gate angles to $[-0.5,0.5]$, chosen after preliminary runs, may itself be responsible for the subtlety of the mutants; repeating the experiment with different angle bounds would show how sensitively the difficulty results depend on this search-space cutoff.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes QUMUG, a search-based approach for generating quantum circuit mutants that pass a given test suite TS while remaining non-equivalent. Mutations are introduced via parameterisable gates (Rx, Ry, Rz, U), with parameters tuned by four search algorithms (GA, HC, (1+1)-EA, random search). The approach is evaluated on 30 quantum circuits (3–7 qubits) drawn from MQT Bench and compared against two existing quantum mutation tools, Muskit and QMutPy. The paper reports that QUMUG mutants are markedly harder to detect, with GA producing on average 494 non-equivalent undetected mutants per program, a 94.3% non-equivalence ratio, and a median of five additional test cases needed to detect them versus one for the baselines. It also presents the first study of higher-order (second-order) quantum mutants, analysing their characteristics and test-case interactions.

Significance. If the empirical claims were fully supported, this would be a valuable contribution: it introduces a new family of parametric mutation operators, provides the first implementation and analysis of higher-order quantum mutants, uses a reasonable experimental design with external baselines, ten repeated runs, statistical testing with effect sizes, and a public repository. The central comparative claim, however, rests on the non-equivalence validation, and that validation currently has a load-bearing multiple-comparisons problem. The approach and benchmarks are promising, but the headline numbers and the comparative conclusion need to be re-established with a sound equivalence criterion before the paper's central claim can be accepted.

major comments (2)
  1. [Section 5.4 / Definition 5] The non-equivalence validation has a severe multiple-comparisons problem that undermines the central claim. A mutant is labelled non-equivalent if at least one test case from CTS yields p<0.05 on the chi-square test. For a truly equivalent mutant, each test case is a Bernoulli trial with a 5% false-positive rate, so the probability of being misclassified as non-equivalent is 1−(0.95)^{|CTS|}. Even under the smallest consistent reading of the CTS size, this is about 71% for 3-qubit circuits and above 99% for 5-qubit circuits, and it is effectively 100% for 7-qubit circuits. The reported 94.3% non-equivalence ratio, the non-equivalent mutant counts in Fig. 4, and the RQ1.2/RQ1.3 analyses that filter on 'non-equivalent' are therefore dominated by Type I errors. Section 7.4.1 discusses only CTS incompleteness (false negatives) and does not address this false-positive problem. The authors should apply a multiple-testing correction (e.g., Benjamini-Hochberg) or use an alternative equivalence criterion (e.g., a minimum Hellinger distance or a joint test across CTS) and re-run the affected analyses.
  2. [Section 5.6] The mutation angle search range was reduced from [-180,180] to [-0.5,0.5] after the authors observed that the vast majority of selected angles in their own preliminary search fell in that range. Because this range is derived from the same search process being evaluated and is used by all QUMUG variants but not by the fixed-gate baselines, the comparative claim that parameterisable-gate mutants are intrinsically more challenging is confounded by this post hoc design choice. A sensitivity analysis with at least one broader range (e.g., [-5,5] or the full [-180,180]) is needed to show that the reported advantage of QUMUG is not an artefact of restricting the search to small perturbations.
minor comments (5)
  1. [Section 5.3] The formulas for test-suite size yield non-integer values for some circuits: for q=3, 2^q×0.4×3 = 9.6 test cases. Please clarify how rounding is performed and whether the reported sizes are exact or approximate.
  2. [Section 5.4] The statement that CTS is five times larger than TS is inconsistent with the preceding description. With TS using 20% classical plus 20% quantum initialisations, and CTS using the remaining 80% classical initialisations plus 'the same number of quantum initialisations', the implied ratio is either 2.5 or 4, depending on interpretation, not 5. Please correct the text and recompute any quantities that depend on this ratio.
  3. [Table 4] P-values are reported inconsistently: some appear as '0.31' or '0.06' while others appear as '<0.05'. Use a uniform convention, e.g., exact p-values or a single threshold notation, to avoid ambiguity.
  4. [Figure 10 / Section 6.2.3] The labels for cases (5) and (6) are confusing because the text and the caption describe the inclusion relationships in opposite directions. Consider renaming these categories with explicit set equations, such as 'tcd_hom ⊂ tcd_fom' and 'tcd_fom ⊂ tcd_hom'.
  5. [Abstract / Section 6.1.2] The abstract says 'three times more challenging' while Section 6.1.2 says 'detected by three times fewer test cases'. Please state the precise metric (detection ratio) in both places so the claim is unambiguous.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: QUMUG's reported mutant difficulty is an empirical comparison against external baselines; self-citations are motivational, and the CTS-based non-equivalence label is an acknowledged operational definition applied uniformly.

full rationale

The paper's central derivation is an empirical search-and-evaluate loop: QUMUG tunes parametric mutation angles to pass TS while maximizing Hellinger distance (Eq. 2), and the reported difficulty metrics are then measured on the larger, independently constructed CTS. No reported headline quantity is a fitted parameter renamed as a prediction. The non-equivalence label is operational: 'A generated mutant is considered non-equivalent if it is detected by at least one test case from CTS' (Section 5.4), so the 94.3% non-equivalent ratio is a CTS-detection ratio by definition. This is a construct-validity choice, not a circular derivation, because the same definition is applied symmetrically to QUMUG, Muskit, and QMutPy, and the paper explicitly acknowledges in Section 7.4.1 that semantic equivalence is not proved. Citations to the authors' earlier work [29, 30, 45] motivate the problem statement and the choice of chi-square testing, but the paper re-establishes the baseline behavior empirically (Fig. 5, Table 5), so those self-citations are not load-bearing. The multiple-testing concern about per-test alpha = 0.05 across the CTS suite is a statistical-validity criticism rather than a circularity of derivation, and it does not change the fact that the generation results are not constructed from the evaluation metric: the search never optimizes against CTS. Accordingly, no step in the claimed derivation chain reduces to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central empirical claim rests on the operational equivalence definition, simulator-based detection, and several experimental design choices such as the angle range and GA settings. These are transparently reported, but they are not derived from first principles and affect the reported numbers.

free parameters (3)
  • Mutation angle search range = [-0.5, 0.5] degrees
    Section 5.6 restricts the search space to one degree after observing that most non-detected mutants found in a preliminary experiment had angles in this range. This hand-chosen restriction directly shapes which mutants QUMUG can find.
  • GA hyperparameters = population size 100, mutation probability 0.5, crossover probability 0.5, 100 generations
    Section 5.6 reports these values were obtained by grid search in a preliminary experiment on small circuits. The reported GA performance depends on these settings.
  • Test suite sampling fraction = 20% of classical inputs plus an equal number of quantum inputs
    Section 5.3 says this percentage was chosen based on preliminary experiments. It determines the strength of TS and therefore which mutants are counted as undetected.
assumptions (4)
  • domain assumption CTS detection is a sufficient proxy for behavioral non-equivalence
    Section 5.4 defines non-equivalent mutants as those detected by CTS, but CTS contains only all classical inputs, a finite set of quantum initialisations, and three measurement bases. Semantic equivalence is not proven.
  • domain assumption Ideal circuit simulation output distributions faithfully represent quantum program behavior
    All experiments run on the Qiskit simulator rather than real hardware. Section 7.4.3 states that evaluating on real noisy devices is currently infeasible for this approach.
  • domain assumption The chi-square test with the chosen shot count reliably distinguishes output distributions
    Section 5.6 uses chi-square at alpha 0.05 with 2^qubits times 2 shots. For larger qubit counts, many outcome bins have small expected counts, so detection classifications may be unreliable.
  • domain assumption Mutants are a valid proxy for real quantum software faults
    Section 7.4.1 acknowledges that the generated mutants are fault-representative testing artefacts rather than exact replications of real-world quantum programming faults.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Search-Based Generation of Undetected Quantum Circuit Mutants." pith.science (2026). https://pith.science/paper/55LUW5CR

@misc{pith2026260809318,
  author       = {Pith},
  title        = {Pith review of: Search-Based Generation of Undetected Quantum Circuit Mutants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55LUW5CR}},
  note         = {Machine review of arXiv:2608.09318}
}
read the original abstract

Quantum mutation analysis is emerging as an essential technique for evaluating test suites due to the limited availability of real faulty quantum programs. However, existing quantum mutation analysis tools use fixed gate-based mutations, resulting in easy-to-detect mutants, which reduces their effectiveness in assessing the quality of test suites. We propose QUMUG, a search-based approach for generating challenging mutants by utilising parameterisable quantum gates. QUMUG employs search algorithms to optimise mutation parameters and find non-equivalent mutants passing a given test suite. In our evaluation over 30 quantum programs, QUMUG produced mutants that are three times more challenging than the mutants generated by existing tools. Among the four evaluated search algorithms, the genetic algorithm was the most effective, generating an average of 494 undetected mutants per program with a 99.67% success rate and 94.3% non-equivalent ratio. The generated mutants demonstrated their effectiveness by requiring the addition of five times more test cases to the test suite than the mutants generated by existing tools. We also analysed the behaviour of higher order mutants in quantum circuits, and showed that while first order mutations are more effective for enhancing the test suite, higher order mutants highlight the need for new unique test cases.

Figures

Figures reproduced from arXiv: 2608.09318 by the authors.

Figure 1
Figure 1. Bloch sphere qubit representation [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Quantum circuit representation of a GHZ Entangled state of three qubits [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Quantum Circuit Operators 2.3 Quantum Mutation Analysis Similar to other software testing techniques, mutation analysis also needs adaptation for quantum software due to its unique characteristics. To facilitate this adaptation, two quantum mutation analysis tools have been developed in the literature, Muskit [29] and QMutPy [13]. These tools treat quantum circuits as the original programs and modify them by applyin… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: RQ1.1 – Median number of mutants passing and non-equivalent by search across all circuits per run. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: RQ1.2 – DetectionRatio for the mutants generated by each approach [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: RQ1.3 – New tc needed to detect mutants generated. indicates that the mutants that pass TS and are not equivalent across approaches have similar DetectionRatio. Concluding remarks for RQ1.2: Even though no big differences were observed among the search-based approaches…
Figure 7
Figure 7. Figure 7: RQ2.1 – Comparison of first order and second order mutants generated by GA. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: RQ2.2 – Qubits and positions utilised for generating second order mutants. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: RQ2.2 – Gate combinations utilised for generating second order mutants. [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: RQ2.3 – Distribution of second order mutants by the relationship between [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 17 canonical work pages

  1. [1]

    Rui Abreu, Shaukat Ali, Paolo Arcaini, José Campos, Michael Felderer, Claude Gravel, Fuyuki Ishikawa, Stefan Klikovits, Andriy Miranskyy, Anila Mjeda, Mohammad Reza Mousavi, Masaomi Yamaguchi, Lei Zhang, and Jianjun Zhao. 2026. Software Testing in the Quantum World.Computer59, 4 (2026), 135–138. doi:10.1109/MC.2026.3655854

    arXiv 2026

  2. [2]

    Rui Abreu, João Paulo Fernandes, Luis Llana, and Guilherme Tavares. 2022. Metamorphic Testing of Oracle Quantum Programs. In2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE). 16–23. doi:10.1145/ 3528230.3529189

    arXiv 2022

  3. [4]

    Paul Ammann, Marcio Eduardo Delamaro, and Jeff Offutt. 2014. Establishing Theoretical Minimal Sets of Mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 21–30. doi:10.1109/ICST. 2014.13

    doi:10.1109/icst 2014

  4. [5]

    1996.Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms

    Thomas Back. 1996.Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press

    work page 1996

  5. [6]

    José Campos and André Souto. 2021. QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments. In2021 IEEE/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE). IEEE Computer Society, Los Alamitos, CA, USA, 28–32. doi:10.1109/Q-SE52...

    arXiv 2021

  6. [7]

    Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions.Psychological bulletin114, 3 (1993), 494. doi:10.1037/0033-2909.114.3.494

    doi:10.1037/0033-2909.114.3.494 1993

  7. [8]

    Cruz-Lemus

    Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and José A. Cruz-Lemus. 2022. Quantum Software Testing: Current Trends and Emerging Proposals. Springer International Publishing, Cham, 167–191. doi:10.1007/978-3-031-05324-5_9

    work page doi:10.1007/978-3-031-05324-5_9 2022

  8. [9]

    Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and Mario Piattini. 2023. Quantum software testing: State of the art.Journal of Software: Evolution and Process35, 4 (2023), e2419. doi:10.1002/smr.2419

    work page doi:10.1002/smr.2419 2023

Show all 57 references
  1. [10]

    Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets.J. Mach. Learn. Res.7 (Dec. 2006), 1–30

    2006

  2. [11]

    Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit.IEEE Transactions on Quantum Engineering3 (2022), 1–17. doi:10.1109/TQE.2022.3195061

    2022

  3. [12]

    Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs Written in QISKit. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computin...

    2022 doi

  4. [13]

    Daniel Fortunato, José Campos, and Rui Abreu. 2022. QMutPy: A Mutation Testing Tool for Quantum Algorithms and Applications in Qiskit. InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis(Virtual, South Korea)(ISSTA 2022). Association ...

    2022

  5. [14]

    Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi. 2026. Robust Mutation Analysis of Quantum Programs Under Noise. arXiv:2605.13279 [cs.SE] https://arxiv.org/abs/2605.13279

    2026 arXiv

  6. [15]

    Stanton A. Glantz. 2012.Primer of Biostatistics(7th edition ed.). McGraw Hill, New York. https://www.accessscience. com/content/book/9780071781503

    2012

  7. [16]

    Greenberger, Michael A

    Daniel M. Greenberger, Michael A. Horne, and Anton Zeilinger. 1989. Going Beyond Bell’s Theorem. InBell’s Theorem, Quantum Theory and Conceptions of the Universe, Menas Kafatos (Ed.). Springer Netherlands, Dordrecht, 69–72. doi:10.1007/978-94-017-0849-4_10

    1989 doi

  8. [17]

    Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property-Based Testing of Quantum Programs in Q#. InProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea)(ICSEW’20). Association for Computin...

    2020

  9. [18]

    Yuta Ishimoto, Masanari Kondo, Naoyasu Ubayashi, Yasutaka Kamei, Ryota Katsube, Naoto Sato, and Hideto Ogawa

  10. [19]

    Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing.IEEE Transactions on Software Engineering37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62

    2011 doi

  11. [20]

    Ryoichi Kawai. 2022. A First Step to Quantum Computation with Qiskit. https://kawaihome.link/jbooks/qcomp- short/cover.html

    2022

  12. [21]

    Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, and Nicos Malevris. 2016. Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SC...

    2016 doi

  13. [22]

    Jonas Klamroth, Max Scheerer, and Oliver Denninger. 2025. Detecting and Tolerating Faults in Hybrid Quantum Software Systems Using Architectural Redundancy. InIEEE International Conference on Quantum Software, QSW 2025, Helsinki, Finland, July 7-12, 2025, Rong N. Chang, Carl K...

    2025

  14. [23]

    Michael Kölle, Tom Bintener, Maximilian Zorn, Gerhard Stenzel, Leo Sünkel, Thomas Gabor, and Claudia Linnhoff- Popien. 2025. Evaluating Mutation Techniques in Genetic-Algorithm-Based Quantum Circuit Synthesis. InProceedings of the Genetic and Evolutionary Computation Conferenc...

    2025

  15. [24]

    Thomas Laurent, Mike Papadakis, Marinos Kintis, Christopher Henard, Yves Le Traon, and Anthony Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 430–435....

    2017 doi

  16. [25]

    Neilson Carlos Leite Ramalho, Higor Amario de Souza, and Marcos Lordello Chaim. 2025. Testing and Debugging Quantum Programs: The Road to 2030.ACM Trans. Softw. Eng. Methodol.34, 5, Article 155 (May 2025), 46 pages. doi:10.1145/3715106

    2025 doi

  17. [26]

    Gushu Li, Li Zhou, Nengkun Yu, Yufei Ding, Mingsheng Ying, and Yuan Xie. 2020. Projection-based runtime assertions for testing and debugging Quantum programs.Proc. ACM Program. Lang.4, OOPSLA, Article 150 (nov 2020), 29 pages. doi:10.1145/3428218

    2020 doi

  18. [27]

    Peixun Long and Jianjun Zhao. 2024. Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing.ACM Trans. Softw. Eng. Methodol.33, 6, Article 147 (June 2024), 61 pages. doi:10.1145/3656339

    2024 doi

  19. [28]

    Kane Meissel and Esther S Yao. 2024. Using Cliff’s delta as a non-parametric effect size measure: an accessible web app and R tutorial.Practical Assessment, Research, and Evaluation29, 1 (2024). doi:10.7275/pare.1977

    2024 doi

  20. [30]

    Eñaut Mendiluze Usandizaga, Shaukat Ali, Tao Yue, and Paolo Arcaini. 2025. Quantum circuit mutants: Empirical analysis and recommendations.Empirical Software Engineering30, 4 (16 Apr 2025), 100. doi:10.1007/s10664-025-10643-z

    2025 doi

  21. [31]

    Search-Based Generation of Undetected Quantum Circuit Mutants

    Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali. 2026. Supplementary material for the paper “Search-Based Generation of Undetected Quantum Circuit Mutants”. https://doi.org/10.5281/zenodo.21718539

    2026 doi

  22. [32]

    Asmar Muqeet, Shaukat Ali, and Paolo Arcaini. 2024. Quantum Program Testing Through Commuting Pauli Strings on IBM’s Quantum Computers. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering(Sacramento, CA, USA)(ASE ’24). Association for ...

    2024

  23. [33]

    Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2024. Mitigating Noise in Quantum Software Testing Using Machine Learning.IEEE Transactions on Software Engineering50, 11 (2024), 2947–2961. doi:10.1109/TSE.2024.3462974

    2024

  24. [34]

    Juan Manuel Murillo, Jose Garcia-Alonso, Enrique Moguel, Johanna Barzen, Frank Leymann, Shaukat Ali, Tao Yue, Paolo Arcaini, Ricardo Pérez-Castillo, Ignacio García-Rodríguez de Guzmán, Mario Piattini, Antonio Ruiz-Cortés, Antonio Brogi, Jianjun Zhao, Andriy Miranskyy, and Manu...

    2025 doi

  25. [35]

    Nielsen and Isaac L

    Michael A. Nielsen and Isaac L. Chuang. 2016.Quantum Computation and Quantum Information (10th Anniversary edition). Cambridge University Press, Cambridge, UK

    2016

  26. [36]

    Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275–378. doi:10.1016/bs.adcom. 2018.03.015

    2019 doi

  27. [37]

    Mike Papadakis and Nicos Malevris. 2010. An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In2010 Third International Conference on Software Testing, Verification, and Validation Workshops. 90–99. doi:10.1109/ICSTW.2010.50

    2010 doi

  28. [38]

    Gabriel Pontolillo, Asmar Muqeet, Shaukat Ali, and Mohammad Reza Mousavi. 2025. From Ideal to Noisy: Adapting Property-Based Testing for Real-World Noisy Quantum Computers. In2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Vol. 01. 405–416. doi:1...

    2025

  29. [39]

    Nils Quetschlich, Lukas Burgholzer, and Robert Wille. 2023. MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing.Quantum7 (2023), 1062. arXiv:2204.13719 doi:10.22331/q-2023-07-20-1062 MQT Bench is available at https://mqt-bench.app/

    2023 arXiv

  30. [40]

    Troels F Rønnow, Zhihui Wang, Joshua Job, Sergio Boixo, Sergei V Isakov, David Wecker, John M Martinis, Daniel A Lidar, and Matthias Troyer. 2014. Defining and detecting quantum speedup.Science345, 6195 (2014), 420–424. doi:10.1126/science.1252319

    2014 doi

  31. [41]

    David Schuler and Andreas Zeller. 2013. Covering and uncovering equivalent mutants.Software Testing, Verification and Reliability23, 5 (2013), 353–374

    2013

  32. [42]

    Birgit Schwarz, David Schuler, and Andreas Zeller. 2011. Breeding High-Impact Mutations. In2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 382–387. doi:10.1109/ICSTW.2011.56

    2011 doi

  33. [43]

    Rodolfo Adamshuk Silva, Simone do Rocio Senger de Souza, and Paulo Sérgio Lopes de Souza. 2017. A systematic review on search based mutation testing.Information and Software Technology81 (2017), 19–35. doi:10.1016/j.infsof.2016.01.017

    2017 doi

  34. [44]

    Sánchez, José A

    Ana B. Sánchez, José A. Parejo, Sergio Segura, Amador Durán, and Mike Papadakis. 2024. Mutation Testing in Practice: Insights From Open-Source Software Developers.IEEE Transactions on Software Engineering50, 5 (2024), 1130–1143. doi:10.1109/TSE.2024.3377378

    2024

  35. [45]

    Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali. 2026. Quantum Circuit Repair by Gate Prioritisation. In2026 IEEE International Conference on Software Testing, Verification and Validation (ICST). 744–748. doi:10.1109/ICST69053.2026.00095

    2026

  36. [46]

    Willem Visser. 2016. What makes killing a mutant hard. InProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering(Singapore, Singapore)(ASE ’16). Association for Computing Machinery, New York, NY, USA, 39–44. doi:10.1145/2970276.2970345

    2016

  37. [47]

    Jiyuan Wang, Fucheng Ma, and Yu Jiang. 2021. Poster: Fuzz Testing of Quantum Program. In2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 466–469. doi:10.1109/ICST49551.2021.00061

    2021

  38. [48]

    Xinyi Wang, Shaukat Ali, and Davide Taibi. 2025. The Landscape of Quantum Software Testing Tools.IEEE Software 42, 5 (2025), 136–140. doi:10.1109/MS.2025.3578154

    2025

  39. [49]

    Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Application of Combinatorial Testing to Quantum Programs. In2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). 179–188. doi:10.1109/QRS54544.2021.00029

    2021

  40. [50]

    Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Generating Failing Test Suites for Quantum Programs With Search. InSearch-Based Software Engineering, Una-May O’Reilly and Xavier Devroey (Eds.). Springer International Publishing, Cham, 9–25. doi:10.1007/978-3-030-88106-1_2

    2021 doi

  41. [53]

    Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search. InProceedings of the Genetic and Evolutionary Computation Conference(Boston, Massachusetts)(GECCO ’22). Association for Comp...

    2022

  42. [54]

    Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods.Biometrics Bulletin1, 6 (1945), 80–83. http: //www.jstor.org/stable/3001968

    1945

  43. [55]

    Woodward

    M.R. Woodward. 1993. Mutation testing—its origin and evolution.Information and Software Technology35, 3 (1993), 163–169. doi:10.1016/0950-5849(93)90053-6

    1993 doi

  44. [56]

    2008.Quantum computing for computer scientists

    Noson S Yanofsky and Mirco A Mannucci. 2008.Quantum computing for computer scientists. Cambridge University Press

    2008

  45. [57]

    Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2024. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering(Ec...

    2024

  46. [58]

    Jianjun Zhao. 2020. Quantum Software Engineering: Landscapes and Horizons.CoRRabs/2007.07047 (2020). arXiv:2007.07047 https://arxiv.org/abs/2007.07047

    2020 arXiv

  47. [59]

    Pengzhan Zhao, Jianjun Zhao, Zhongtao Miao, and Shuhan Lan. 2021. Bugs4Q: A Benchmark of Real Bugs for Quantum Programs. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1373–

    2021

  48. [1376]

    doi:10.1109/ASE51524.2021.9678908 ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026

    2021

  49. [2025]

    InProceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE ’25)

    Evaluating Mutation-based Fault Localization for Quantum Programs. InProceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE ’25). Association for Computing Machinery, New York, NY, USA, 666–671. doi:10.1145/3756681.3757022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.