Pith. sign in

REVIEW 2 major objections 4 minor 87 references

On the Feasibility of Quantum Unit Testing

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new quantum unit test that appends the inverse of the expected state to a circuit distinguishes faulty from fault-free circuits with fewer measurements than the standard Swap test.

desk verdict A serious, large-scale study of quantum unit tests whose headline shot-efficiency result is partly biased by an Inverse-anchored shot cap; worth refereeing but needs major revision. read the letter →

arxiv 2507.17235 v1 pith:GGUTTUVM submitted 2025-07-23 cs.SE quant-ph

Andriy Miranskyy , José Campos , Anila Mjeda , Lei Zhang , Ignacio García Rodríguez de Guzmán This is my paper · ORCID
classification cs.SEquant-ph
keywords quantumunittestingInversetestSwapStatevectorChernoffBoundmutationsoftwareengineeringfault-tolerantcomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that quantum unit testing is feasible and that a newly proposed Inverse test is the most shot-efficient executable test among the alternatives. The test works by appending the conjugate transpose of the expected state to the circuit under test: if the circuit is correct, the final measurement is always the all-zero bitstring, and any other outcome is proof of a fault. The authors evaluate nine tests on 1.8 million mutated quantum circuits and report that the Inverse test needs 60–66% fewer shots than the Swap test and orders of magnitude fewer than statistical tests, while never producing false positives on noiseless hardware. If these results carry to fault-tolerant machines, quantum software developers gain a cheap, deterministic-style unit oracle.

What carries the argument

The load-bearing object is the Inverse-test circuit: prepare the input state, run the program under test, append the conjugate transpose of the expected-state preparation, and measure; the all-zero bitstring is the unique pass signature. Its efficiency is guaranteed by the Quantum Chernoff Bound, which for this test reduces to a shot-count formula $N \sim \ln(P_e)/\ln(\min_{0\le s\le1}\operatorname{Tr}(\rho^s\sigma^{1-s}))$, and in the diagonal case to $N = \max(\lceil\ln(P_e)/\ln(\sigma_{11})\rceil, 1)$. This formula both predicts the exponential growth in shots as the faulty state approaches the correct one and gives practitioners a concrete stopping rule, which the empirical study confirms is slightly conservative.

What would settle it

Run the Inverse test on a correct (unmutated) 5-qubit circuit on a noisy device or in a simulator with a realistic depolarizing error model (e.g., $10^{-3}$ per gate) for 10,000 shots; if a non-zero bitstring is observed with probability above the nominal $P_e = 0.05$, then the zero-false-positive guarantee fails on that hardware, and if the shots needed to detect a fixed one-degree rotation error then exceed the Swap test's requirement, the ranking reverses.

Watch

Extended reading notes

Core claim

The central discovery is that a reversibility check makes a practically usable unit test: encode the expected behaviour into the test circuit itself, and a single measurement pattern (all zeros) means 'pass', while any deviation is a guaranteed fail. In the authors' formulation the Inverse test prepares the input state, applies the program under test, then applies the conjugate transpose of the operation that would produce the expected state; on a correct circuit this collapses the output to the zero-state with probability one. Using the Quantum Chernoff Bound the paper derives a shot-count rule $N \sim \ln(P_e)/\ln(\min_{0\le s\le1}\operatorname{Tr}(\rho^s\sigma^{1-s}))$, which for a diagonal faulty state becomes $\lceil\ln(P_e)/\ln(\sigma_{11})\rceil$ and grows exponentially as the faulty state approaches the correct one. Empirically, over 1,796,880 mutated circuits, the Inverse test was rank-1 in 69%–78% of cases with median shot counts of 2–3, versus 5–9 for the Swap test and roughly 600–1900 for statistical tests; its recall was 0.904 on random circuits and 0.975 on benchmark circuits, with zero false positives by construction.

Load-bearing premise

All performance claims assume a fault-tolerant quantum computer or an ideal noise-free simulator; if real hardware noise makes the inverse circuit's extra depth create spurious failures, the zero-false-positive guarantee and the shot-count advantage break down.

Editorial extensions

If this is right

  • If the Inverse test is adopted on fault-tolerant hardware, verifying a quantum subroutine could cost 60–66% fewer shots than with the Swap test, directly reducing quantum compute time.
  • Because it produces no false positives on noiseless execution, a passing Inverse test gives a developer certainty that the circuit matches the expected state, eliminating the flaky-test behaviour that statistical tests exhibit.
  • The Quantum Chernoff Bound formula supplies a rule for choosing the number of shots, and the observed 3.3% missed-fault rate at a 5% error budget suggests the formula is safe, if slightly conservative.
  • The formal Arrange–Act–Assert definition of a quantum unit test gives a reusable template: any oracle that recognizes the expected state can be wrapped in the same harness.
  • For subroutines whose expected states are known, testing small instances (e.g., 1–5 qubits) with a deterministic oracle like the Statevector or Inverse test can cover the main execution paths without full-state simulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the shot-count advantage persists on fault-tolerant devices, quantum test suites could be embedded in continuous-integration pipelines with per-subroutine inverse checks, making quantum software engineering resemble classical unit testing more closely than current sampling-based oracles allow.
  • Editorial extension: the inverse check could serve as a mid-circuit assertion to localize faults after a subroutine, though the paper only treats it as an end-of-circuit unit test; the extra depth and measurement disruption are the natural costs to investigate.
  • Editorial extension: Equation (8) is derived for diagonal faulty states; a numerical per-circuit evaluation for non-diagonal and mixed states would produce adaptive shot budgets, a direct generalization the paper leaves open.
  • Editorial caution: since the Inverse test's circuit is roughly twice as deep as the statistical tests', on near-term noisy hardware gate errors could create false positives and overturn the shot-count ranking; comparing the tests under a realistic noise model is the immediate next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper formalizes quantum unit tests for four families—statistical tests, the Swap test, the Statevector test, and a new Inverse test—and compares them both theoretically and empirically. The theoretical part derives a shot-count estimate for the Inverse test from the Quantum Chernoff Bound, and the empirical part applies mutation operators to 10,000 random circuits and 85 MQT Bench circuits, yielding 1,796,880 mutated circuits that are tested with nine methods under a shared shot cap. The paper reports that the Inverse test performs best at distinguishing different quantum states (RQ1) and requires 60--66% fewer shots than the Swap test (RQ2), with the Statevector test being exact but not scalable.

Significance. If the central claims hold, the paper is a valuable contribution to quantum software testing: it gives a clean formal definition of a quantum unit test, introduces a novel reversible-circuit-based Inverse test, derives a parameter-free shot-count prediction from the Quantum Chernoff Bound, and evaluates on an unusually large corpus (1.8M mutated circuits, 100 repetitions per experiment). The paper is also careful to state its fault-tolerant/noiseless execution assumption. However, the headline empirical claims are currently weakened by two technical problems: an incorrect derivation in the Case 1 analysis of Section III-B4, and an asymmetric shot budget in Section IV-C1b that favors the Inverse test over the Swap test. Because RQ1 and RQ2 depend on exactly those comparisons, the conclusions need to be revisited with a fair budgeting scheme.

major comments (2)
  1. [Section III-B4, Case 1 (Eq. (5))] The analysis of Case 1 states that for a diagonal sigma with sigma_11 = 0, Tr(rho sigma^{1-s}) = 1. This is false for s < 1: the (1,1) entry of sigma^{1-s} is 0^{1-s} = 0, so the trace is 0, with the usual convention 0^0 = 1 applying only at s = 1. The correct minimum is 0, which gives xi_QCB = +infinity and an error probability of 0, not Pe ~ exp(-N * 0) = 1. The displayed equation is also internally inconsistent with the following sentence claiming that a single shot suffices: if the error probability were 1, no number of shots would detect the difference. The qualitative conclusion (one shot detects orthogonal states) is correct, but the derivation and the displayed formula need to be corrected.
  2. [Section IV-C1b and Table III / RQ2] The shot budget is capped at twice the theoretical limit given by Equation (8), which is the Inverse test's own shot-count formula, and this same cap is applied to all tests. For a mutant with fidelity f between the actual and expected pure states, the Inverse test's per-shot false-negative probability is f, while the Swap test's is (1+f)/2. Since ((1+f)/2)^2 > f for every f < 1, the number of shots the Swap test needs to reach the same Pe = 0.05 is strictly greater than twice the Inverse test's requirement. The cap of 2N_inv therefore denies the Swap test a budget sufficient for its own target error rate, while granting the Inverse test its target budget. Consequently, the recall differences in Table III (Inverse 0.904 vs Swap 0.824 on Random Circuits; 0.975 vs 0.958 on MQT Bench) and the median shot counts in Figure 8 partly reflect this asymmetric, Inverse-anchored budget rather than intrinsic test performance. This directly weakens the RQ2 claim that the Inverse test 'keeps false negatives low' relative to the Swap test. The comparison should give each test its own theoretical budget for a common error probability, or explicitly analyze the censored results.
minor comments (4)
  1. [Section IV-C1c] The stated total number of experiments, 1,585,665,000 = (1,751,850 + 10,000) pairs x 9 tests x 100 repetitions, does not match the reported total of 1,796,880 mutated circuits (1,751,850 random plus 45,030 MQT Bench). Please correct the count and clarify whether the deterministic Statevector test is included in the 100 repetitions.
  2. [Section III-B4, Case 2] The displayed derivative of f(sigma_11, s) = ln(sigma_11^{1-s}) is garbled; the correct derivative with respect to s is -ln(sigma_11), which is never zero for sigma_11 in (0,1). The final conclusion that the minimum occurs at s = 0 is correct, but the intermediate expression should be fixed.
  3. [Section I, Table I] The Multinomial test rows appear to contain formatting or entry errors, such as totals of 2000 in a 200-repetition experiment and duplicated TP/TN values. Please verify the table entries.
  4. [Section IV-C1b and Figure 8] The global caps (10^4 shots for Random Circuits and 10^6 for MQT Bench) are orders of magnitude below some theoretical shot counts shown in Figure 5; since Figure 8 excludes outliers and false negatives, the censoring mechanism and its effect on the reported medians should be described more explicitly.

Circularity Check

2 steps flagged · score 6.0 of 10

Shot cap derived from the Inverse test's own Equation (8) is applied to all tests, making the shot-count and recall comparisons and the 3.3% false-negative validation partly self-fulfilling.

  1. self definitional [Section IV-C1b; used in Section IV-D (RQ2, Table III, Figure 8)]
    "For individual experiments (i.e., a test on a pair original-mutant), the number of shots is capped at twice the theoretical limit given by Equation (8) with Pe = 0.05. This cap is applied across all tests, although the analysis focuses specifically on the Inverse test."

    Let f be the fidelity between the actual and expected states. For the Inverse test, a shot fails to detect the bug with probability f, so the shots needed for Pe=0.05 are N_inv = ln(0.05)/ln(f). For the Swap test, the ancilla stays 0 with probability (1+f)/2, so its needed shots are N_swap = ln(0.05)/ln((1+f)/2). Since ((1+f)/2)^2 > f for every 0<f<1, N_swap > 2*N_inv. Hence the cap of 2*N_inv is strictly smaller than the Swap test's own sufficient budget for every non-identical mutant. Applying this cap to all tests inflates the Swap test's false negatives and censors its shot counts, so the Table III recall gap and RQ2's '60% to 66% fewer shots' conclusion are partly constructed by Inverse-test theory rather than an independent comparison.

  2. self definitional [Section IV-D, paragraph after Table III]
    "our analysis indicates that about 3.3% of cases expected to be flagged were missed, slightly below the anticipated 5% (as Pe = 0.05). This implies that the theoretical formula for required shot count from Equation (8), aligns well with empirical results and may even be slightly conservative in practice."

    The cap is 2*N_inv with N_inv from Equation (8) at Pe=0.05; under that cap the Inverse test is given twice its theoretical requirement, so its false-negative rate is by design around or below the Pe used in the formula. The observed 3.3% missed cases is therefore the designed operating point of the experiment, not an independent empirical check of Equation (8). The statement that Equation (8) 'aligns well with empirical results and may even be slightly conservative' follows from using the same equation to set the budget.

full rationale

The derivation of Equation (8) from the Quantum Chernoff Bound is self-contained and parameter-free, and the zero-false-positive properties of the Inverse and Swap tests are structural; those parts are not circular. The circularity is in the evaluation protocol. Section IV-C1b sets every test's shot budget to twice the Inverse test's own theoretical requirement from Equation (8). For any non-identical mutant with fidelity f, the Swap test needs strictly more than twice as many shots as the Inverse test to reach the same Pe, so the Swap test is systematically under-budgeted. The observed recall ranking (Inverse 0.904 vs Swap 0.824 on Random Circuits) and the median shot-count ranking in RQ2 are therefore partly forced by the Inverse test's own formula rather than measured on equal terms. In addition, the paper offers the 3.3% false-negative rate as independent confirmation of Equation (8), but that rate is the designed operating point of a cap computed from Equation (8); it is not an external check. This is a partial, not total, circularity: the theory itself is externally grounded (QCB), and the Statevector-test comparison and structural false-positive analysis remain independent. Score 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The study's conclusions rest on standard quantum-mechanical unitarity, the Quantum Chernoff Bound, exact knowledge of expected states, a chosen set of mutation operators, and small test circuits. None of these are derived in the paper; they are imported as assumptions. The Inverse test is a circuit construction, not a new physical entity, so no invented entities are listed.

free parameters (5)
  • Significance threshold Pe for shot-count cap = 0.05
    Hand-chosen target error probability used in Equation (8) to set the shot cap; changing it changes required shots and which mutants are missed.
  • Shot-count cap multiplier = 2x theoretical S
    Section IV-C1b caps shots at twice the theoretical limit; this choice moderates the recall comparison and is not derived from first principles.
  • Global shot cap = 1e4 (Random Circuits), 1e6 (MQT Bench)
    Chosen to bound CPU time; directly affects false-negative rates and ranking results.
  • Number of Monte Carlo repetitions M = 1000
    Section IV-C1d; sets the standard error of empirical p-values to roughly 0.007.
  • RGI rotation angle = theta = phi = pi/180
    One-degree rotation mutation strength; controls how subtle inserted faults are in the fault model.
assumptions (5)
  • domain assumption Quantum circuits are perfectly unitary and reversible, with no decoherence or gate errors.
    Used in Sections III-B2 and III-B4 to argue Swap and Inverse tests have zero false positives; Section IV-E3 states execution on FT or noise-free simulator.
  • domain assumption The expected output |psi_E> is known exactly and can be synthesized or inverted as a circuit Z.
    Section III-A3b acknowledges the oracle-problem limitation; Algorithm 5 requires Z with Z|psi_E>=|0>.
  • standard math The Quantum Chernoff Bound from Audenaert et al. and Nussbaum-Szkola is the correct error-exponent tool.
    Equations (2) and (8) rely on this external result to predict shot counts.
  • domain assumption Mutation operators QGR, QGD, QGI, and RGI represent realistic developer faults.
    Section IV-A adopts operators from Fortunato et al. and Mendiluze et al.; no real-world quantum bug dataset is used (QBugs unavailable, Bugs4Q has 20 bugs).
  • domain assumption The 10,000 random circuits plus 85 MQT Bench circuits with n<=5 generalize to larger quantum programs.
    Section IV-B caps n at 5 due to exponential simulation cost; Section IV-E3 lists this as an external validity threat.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Feasibility of Quantum Unit Testing." pith.science (2026). https://pith.science/paper/GGUTTUVM

@misc{pith2026250717235,
  author       = {Pith},
  title        = {Pith review of: On the Feasibility of Quantum Unit Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGUTTUVM}},
  note         = {Machine review of arXiv:2507.17235}
}
read the original abstract

The increasing complexity of quantum software presents significant challenges for software verification and validation, particularly in the context of unit testing. This work presents a comprehensive study on quantum-centric unit tests, comparing traditional statistical approaches with tests specifically designed for quantum circuits. These include tests that run only on a classical computer, such as the Statevector test, as well as those executable on quantum hardware, such as the Swap test and the novel Inverse test. Through an empirical study and detailed analysis on 1,796,880 mutated quantum circuits, we investigate (a) each test's ability to detect subtle discrepancies between the expected and actual states of a quantum circuit, and (b) the number of measurements required to achieve high reliability. The results demonstrate that quantum-centric tests, particularly the Statevector test and the Inverse test, provide clear advantages in terms of precision and efficiency, reducing both false positives and false negatives compared to statistical tests. This work contributes to the development of more robust and scalable strategies for testing quantum software, supporting the future adoption of fault-tolerant quantum computers and promoting more reliable practices in quantum software engineering.

Figures

Figures reproduced from arXiv: 2507.17235 by the authors.

Figure 1
Figure 1. Quantum testing circuit for the Statistical tests. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Quantum testing circuit for the Swap test. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Quantum testing circuit for the Inverse test. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The value of N from Equation (7) for 0.001 ≤ σ11 ≤ 0.999 and Pe ∈ {0.001, 0.01, 0.05}. In practice, we round up N to the nearest integer away from zero, ensuring there is at least one shot: N ∼ max  ln (Pe) ln(σ11)  , 1  , (7) where ⌈·⌉ denotes the ceiling function…
Figure 5
Figure 5. Figure 5: Theoretical shot count S based on Equation (8) with Pe = 0.05. 0 25 50 75 1 2 3 4 5 Number of Qubits Percentage of Rank−1 Test Name Chi−sq. Test G Test Multinomial Test MC Chi−sq. Test MC G Test MC Multinomial Test Swap Test Inverse Test Dataset 1. Random Circuits 2. M…
Figure 6
Figure 6. Figure 6: Variation in the percentage of rank-1 cases with the [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Ranking performance comparison. 1. Random Circuits 2. MQT Bench 1 10 100 1000 10000 1 10 100 1000 10000 Inverse Test Swap Test MC Multinomial Test MC G Test MC Chi−sq. Test Multinomial Test G Test Chi−sq. Test Test Shot Count Test Name [PITH_FULL_IMAGE:figures/full_fi…
Figure 8
Figure 8. Figure 8: Shot count performance comparison (outliers not visualized). False negatives are excluded from the plots. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 34 canonical work pages

  1. [1]

    M. A. Nielsen and I. L. Chuang,Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge Univ. Press, 2010. DOI: 10.1017/CBO9780511976667

    doi:10.1017/cbo9780511976667 2010

  2. [2]

    A survey on unit testing practices and problems,

    E. Daka and G. Fraser, “A survey on unit testing practices and problems,” in2014 IEEE 25th International Symposium on Software Reliability Engineering, IEEE, 2014, pp. 201–211

    2014

  3. [3]

    On testing quantum programs,

    A. Miranskyy and L. Zhang, “On testing quantum programs,” in2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2019, pp. 57–60.DOI: 10.1109/ICSE-NIER.2019.00023

    arXiv 2019

  4. [4]

    Is your quantum program bug-free?

    A. Miranskyy, L. Zhang, and J. Doliskani, “Is your quantum program bug-free?” InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, ser. ICSE-NIER ’20, Association for Computing Machinery, 2020, 29–32.DOI: 10.1145/3377816.3381731

    arXiv 2020

  5. [5]

    Miranskyy, L

    A. Miranskyy, L. Zhang, and J. Doliskani,On testing and debugging quantum software, 2021.DOI: 10.48550/arXiv.2103.09172 arXiv: 2103.09172

    arXiv doi:10.48550/arxiv.2103.09172 2021

  6. [6]

    Toward a quantum software engineering,

    M. Piattini, M. Serrano, R. Perez-Castillo, G. Petersen, and J. L. Hevia, “Toward a quantum software engineering,”IT Professional, vol. 23, no. 1, pp. 62–66, 2021.DOI: 10.1109/MITP.2020.3019522

    work page arXiv 2021

  7. [7]

    The talavera manifesto for quantum software engineering and programming,

    M. Piattini et al., “The talavera manifesto for quantum software engineering and programming,” inShort Papers Proceedings of the 1st International Workshop on the QuANtum SoftWare Engineering & pRogramming, Talavera de la Reina, Spain, February 11-12, 2020, M. Piattini, G. Peterssen, R. P ´erez-Castillo, J. L. Hevia, and M. A. Serrano, Eds., ser. CEUR Wor...

    2020

  8. [8]

    Zhao,Quantum software engineering: Landscapes and horizons,

    J. Zhao,Quantum software engineering: Landscapes and horizons,

Show all 87 references
  1. [9]

    Testing and debugging quantum programs: The road to 2030,

    N. C. L. Ramalho, H. Amario de Souza, and M. Lordello Chaim, “Testing and debugging quantum programs: The road to 2030,”ACM Trans. Softw. Eng. Methodol., Jan. 2025,ISSN: 1049-331X.DOI: 10. 1145/3715106 [Online]. Available: https://doi.org/10.1145/3715106

    2025 doi

  2. [10]

    Quantum software engineering: Roadmap and challenges ahead,

    J. M. Murillo et al., “Quantum software engineering: Roadmap and challenges ahead,”ACM Trans. Softw. Eng. Methodol., vol. 34, no. 5, May 2025.DOI: 10.1145/3712002

    2025 doi

  3. [11]

    Stabilization of quantum computations by symmetrization,

    A. Barenco, A. Berthiaume, D. Deutsch, A. Ekert, R. Jozsa, and C. Mac- chiavello, “Stabilization of quantum computations by symmetrization,” SIAM Journal on Computing, vol. 26, no. 5, pp. 1541–1557, 1997.DOI: 10.1137/S0097539796302452

    1997 doi

  4. [12]

    Quantum fingerprinting,

    H. Buhrman, R. Cleve, J. Watrous, and R. de Wolf, “Quantum fingerprinting,”Phys. Rev. Lett., vol. 87, p. 167 902, 16 2001.DOI: 10.1103/PhysRevLett.87.167902

    2001 doi

  5. [13]

    Quantum circuit mutants: Empirical analysis and recommendations,

    E. n. Mendiluze Usandizaga, S. Ali, T. Yue, and P. Arcaini, “Quantum circuit mutants: Empirical analysis and recommendations,”Empirical Software Engineering, vol. 30, no. 4, Apr. 2025,ISSN: 1382-3256.DOI: 10.1007/s10664-025-10643-z

    2025 doi

  6. [14]

    Discriminating states: The quantum chernoff bound,

    K. M. R. Audenaert, J. Calsamiglia, R. Mu ˜noz-Tapia, E. Bagan, L. Masanes, A. Acin, and F. Verstraete, “Discriminating states: The quantum chernoff bound,”Phys. Rev. Lett., vol. 98, p. 160 501, 16 Apr. 2007.DOI: 10.1103/PhysRevLett.98.160501

    2007 doi

  7. [15]

    The Chernoff lower bound for symmetric quantum hypothesis testing,

    M. Nussbaum and A. Szkoła, “The Chernoff lower bound for symmetric quantum hypothesis testing,”The Annals of Statistics, vol. 37, no. 2, pp. 1040–1057, 2009.DOI: 10.1214/08-AOS593

    2009 doi

  8. [16]

    Pearson, “X

    K. Pearson, “X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling,”The London, Edinburgh, and Dublin Philosophical Magazine and Jou...

    1900 doi

  9. [17]

    J. H. McDonald,Handbook of biological statistics, 3rd ed. Sparky House Publishing, 2014

    2014

  10. [18]

    QuSBT: Search-based testing of quantum programs,

    X. Wang, P. Arcaini, T. Yue, and S. Ali, “QuSBT: Search-based testing of quantum programs,” inProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022, pp. 173–177

    2022

  11. [19]

    QuCAT: A combinatorial testing tool for quantum software,

    X. Wang, P. Arcaini, T. Yue, and S. Ali, “QuCAT: A combinatorial testing tool for quantum software,” in2023 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE), IEEE, 2023, pp. 2066–2069

    2023

  12. [20]

    Quito: A coverage-guided test generator for quantum programs,

    X. Wang, P. Arcaini, T. Yue, and S. Ali, “Quito: A coverage-guided test generator for quantum programs,” in2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 2021, pp. 1237–1241

    2021

  13. [21]

    Identifying flakiness in quantum programs,

    L. Zhang, M. Radnejad, and A. Miranskyy, “Identifying flakiness in quantum programs,” inACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM, IEEE, 2023, pp. 1–7.DOI: 10.1109/ESEM56168.2023.10304850

    2023

  14. [23]

    Automat- ing quantum software maintenance: Flakiness detection and root cause analysis,

    J. Sivaloganathan, A. Jamshidi, A. Miranskyy, and L. Zhang, “Automat- ing quantum software maintenance: Flakiness detection and root cause analysis,”arXiv, 2024.DOI: 10.48550/arXiv.2410.23578

    2024 doi arXiv

  15. [24]

    MQT Bench: Bench- marking software and design automation tools for quantum computing,

    N. Quetschlich, L. Burgholzer, and R. Wille, “MQT Bench: Bench- marking software and design automation tools for quantum computing,” Quantum, 2023, MQT Bench is available at https://www.cda.cit.tum. de/mqtbench/

    2023

  16. [25]

    Statistical assertions for validating patterns and finding bugs in quantum programs,

    Y . Huang and M. Martonosi, “Statistical assertions for validating patterns and finding bugs in quantum programs,” inProceedings of the 46th International Symposium on Computer Architecture, ser. ISCA ’19, Phoenix, Arizona: Association for Computing Machinery, 2019, pp. 541–55...

    2019

  17. [26]

    Quassert: Automatic generation of quantum assertions,

    H. Witharana, D. V olya, and P. Mishra, “Quassert: Automatic generation of quantum assertions,”arXiv preprint arXiv:2303.01487, 2023

    2023 arXiv

  18. [27]

    Quantum circuits for dynamic runtime assertions in quantum computation,

    H. Zhou and G. T. Byrd, “Quantum circuits for dynamic runtime assertions in quantum computation,”IEEE Computer Architecture Letters, vol. 18, no. 2, pp. 111–114, 2019

    2019

  19. [28]

    Quantum circuits for dynamic runtime assertions in quantum computation,

    J. Liu, G. T. Byrd, and H. Zhou, “Quantum circuits for dynamic runtime assertions in quantum computation,” inProceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, 2020, pp. 1017–1030. DOI: 10.1109/LCA...

    2020

  20. [29]

    Exploiting quantum assertions for error mitigation and quantum program debugging,

    P. Li, J. Liu, Y . Li, and H. Zhou, “Exploiting quantum assertions for error mitigation and quantum program debugging,” in2022 IEEE 40th International Conference on Computer Design (ICCD), IEEE, 2022, pp. 124–131

    2022

  21. [30]

    Qraft: reverse your Quantum circuit and know the correct program output,

    T. Patel and D. Tiwari, “Qraft: reverse your Quantum circuit and know the correct program output,” inProceedings of the 26th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS ’21, Virtual, USA: Association for ...

    2021

  22. [31]

    Vanqira: A vanishing-state- based framework for quantum circuit runtime assertion,

    T.-F. Chen, C. -Y . Wei, and J.-H. R. Jiang, “Vanqira: A vanishing-state- based framework for quantum circuit runtime assertion,” in2023 IEEE International Conference on Quantum Computing and Engineering (QCE), IEEE, vol. 1, 2023, pp. 1033–1043

    2023

  23. [32]

    Quantumunit: A proposal for classic multi-qubit assertion development,

    I. G.-R. de Guzm ´an, A. G. de la Barrera Amo, M. ´A. Serrano, M. Polo, and M. Piattini, “Quantumunit: A proposal for classic multi-qubit assertion development,” inConference on Cloud Computing, Big Data & Emerging Topics, Springer, 2024, pp. 121–131

    2024

  24. [33]

    Automatically refining assertions for efficient debugging of quantum programs,

    D. Rovara, L. Burgholzer, and R. Wille, “Automatically refining assertions for efficient debugging of quantum programs,”arXiv preprint arXiv:2412.14252, 2024

    2024 arXiv

  25. [34]

    A framework for debugging quantum programs,

    D. Rovara, L. Burgholzer, and R. Wille, “A framework for debugging quantum programs,”arXiv preprint arXiv:2412.12269, 2024

    2024 arXiv

  26. [35]

    Projection- based runtime assertions for testing and debugging quantum programs,

    G. Li, L. Zhou, N. Yu, Y . Ding, M. Ying, and Y . Xie, “Projection- based runtime assertions for testing and debugging quantum programs,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, Nov. 2020.DOI: 10.1145/3428218 17

    2020 doi

  27. [36]

    Statistical assertions for validating patterns and finding bugs in quantum programs,

    Y . Huang and M. Martonosi, “Statistical assertions for validating patterns and finding bugs in quantum programs,” inProceedings of the 46th International Symposium on Computer Architecture, 2019, pp. 541–553

    2019

  28. [37]

    Systematic approaches for precise and approximate quantum state runtime assertion,

    J. Liu and H. Zhou, “Systematic approaches for precise and approximate quantum state runtime assertion,” in2021 IEEE International Sympo- sium on High-Performance Computer Architecture (HPCA), IEEE, 2021, pp. 179–193

    2021

  29. [38]

    QuraTest: Integrating Quantum Specific Features in Quantum Program Testing,

    J. Ye, S. Xia, F. Zhang, P. Arcaini, L. Ma, J. Zhao, and F. Ishikawa, “QuraTest: Integrating Quantum Specific Features in Quantum Program Testing,” inProceedings of the 38th IEEE/ACM International Confer- ence on Automated Software Engineering, ser. ASE ’23, Echternach, Luxemb...

    2024

  30. [39]

    Pontolillo, M

    G. Pontolillo, M. R. Mousavi, and M. Grzesiuk,QuCheck: A Property- based Testing Framework for Quantum Programs in Qiskit, 2025. arXiv: 2503.22641[quant-ph]

    2025 arXiv

  31. [41]

    An exploratory study on the usage of quantum programming languages,

    F. Ferreira and J. Campos, “An exploratory study on the usage of quantum programming languages,”Science of Computer Programming, vol. 240, p. 103 217, 2025,ISSN: 0167-6423.DOI: https://doi.org/10. 1016/j.scico.2024.103217

    2025

  32. [42]

    Automatic generation of test circuits for the verification of quantum deterministic algorithms,

    A. Garc ´ıa de la Barrera Amo, M. A. Serrano, I. Garc ´ıa Rodr´ıguez de Guzm´an, M. Polo, and M. Piattini, “Automatic generation of test circuits for the verification of quantum deterministic algorithms,” inProceedings of the 1st International Workshop on Quantum Programming f...

    2022

  33. [43]

    How Do Developers Structure Unit Test Cases? An Empirical Analysis of the AAA Pattern in Open Source Projects,

    C. Wei, L. Xiao, T. Yu, S. Wong, and A. Clune, “How Do Developers Structure Unit Test Cases? An Empirical Analysis of the AAA Pattern in Open Source Projects,”IEEE Trans. Softw. Eng., vol. 51, no. 4, 1007–1038, Jan. 2025,ISSN: 0098-5589.DOI: 10.1109/TSE.2025. 3537337

    2025 doi

  34. [44]

    OpenQASM 3: A Broader and Deeper Quantum Assembly Language,

    A. Cross et al., “OpenQASM 3: A Broader and Deeper Quantum Assembly Language,”ACM Transactions on Quantum Computing, vol. 3, no. 3, Sep. 2022.DOI: 10.1145/3505636

    2022 doi

  35. [45]

    Systematic Approaches for Precise and Approxi- mate Quantum State Runtime Assertion,

    J. Liu and H. Zhou, “Systematic Approaches for Precise and Approxi- mate Quantum State Runtime Assertion,” in2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2021, pp. 179–193.DOI: 10.1109/HPCA51647.2021.00025

    2021

  36. [46]

    The Oracle Problem in Software Testing: A Survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The Oracle Problem in Software Testing: A Survey,”IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015.DOI: 10.1109/TSE.2014.2372785

    2015

  37. [47]

    A Comparative Study on Automated Software Test Oracle Methods,

    S. R. Shahamiri, W. M. N. W. Kadir, and S. Z. Mohd-Hashim, “A Comparative Study on Automated Software Test Oracle Methods,” in2009 Fourth International Conference on Software Engineering Advances, 2009, pp. 140–145.DOI: 10.1109/ICSEA.2009.29

    2009 doi

  38. [48]

    Chapter Three - Automated Test Oracles: State of the Art, Taxonomies, and Trends,

    R. A. Oliveira, U. Kanewala, and P. A. Nardi, “Chapter Three - Automated Test Oracles: State of the Art, Taxonomies, and Trends,” in ser. Advances in Computers, A. Memon, Ed., vol. 95, Elsevier, 2014, pp. 113–199.DOI: 10.1016/B978-0-12-800160-8.00003-6

    2014 doi

  39. [49]

    Chapter One - Automated Test Oracles: A Survey,

    M. Pezz `e and C. Zhang, “Chapter One - Automated Test Oracles: A Survey,” in ser. Advances in Computers, A. Memon, Ed., vol. 95, Elsevier, 2014, pp. 1–48.DOI: 10.1016/B978-0-12-800160-8.00001-2

    2014 doi

  40. [50]

    Differential testing for software,

    W. M. McKeeman, “Differential testing for software,”Digital Technical Journal, vol. 10, no. 1, pp. 100–107, 1998

    1998

  41. [51]

    Disposable Testing: Avoiding Maintenance of Generated Unit Tests by Throwing Them Away,

    S. Shamshiri, J. Campos, G. Fraser, and P. McMinn, “Disposable Testing: Avoiding Maintenance of Generated Unit Tests by Throwing Them Away,” in2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), 2017, pp. 207–209.DOI: 10.1109/ICSE-C.2017.100

    2017 doi

  42. [52]

    Perception and Practices of Differential Testing,

    M. A. Gulzar, Y . Zhu, and X. Han, “Perception and Practices of Differential Testing,” in2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE- SEIP), 2019, pp. 71–80.DOI: 10.1109/ICSE-SEIP.2019.00016

    2019

  43. [53]

    Differential testing: a new approach to change detection,

    R. B. Evans and A. Savoia, “Differential testing: a new approach to change detection,” inThe 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers, ser. ESEC- FSE companion ’07, ...

    2007

  44. [54]

    Casella and R

    G. Casella and R. L. Berger,Statistical inference, 2nd ed. Duxbury, 2001

    2001

  45. [55]

    The tale of cochran’s rule: My contingency table has so many expected values smaller than 5, what am i to do?

    P. M. Kroonenberg and A. Verbeek, “The tale of cochran’s rule: My contingency table has so many expected values smaller than 5, what am i to do?”The American Statistician, vol. 72, no. 2, pp. 175–183, 2018.DOI: 10.1080/00031305.2017.1286260

    2018 arXiv

  46. [56]

    Some methods for strengthening the common χ2 tests,

    W. G. Cochran, “Some methods for strengthening the common χ2 tests,”Biometrics, vol. 10, no. 4, pp. 417–451, 1954. [Online]. Available: http://www.jstor.org/stable/3001616

    1954

  47. [57]

    Javadi-Abhari et al.,Quantum computing with Qiskit, 2024.DOI: 10.48550/arXiv.2405.08810 arXiv: 2405.08810

    A. Javadi-Abhari et al.,Quantum computing with Qiskit, 2024.DOI: 10.48550/arXiv.2405.08810 arXiv: 2405.08810

    2024 doi arXiv

  48. [58]

    How to check a given unitary evolution is correct in a real quantum computer in Qiskit? - Quantum Computing Stack Exchange,

    Q. Wang. “How to check a given unitary evolution is correct in a real quantum computer in Qiskit? - Quantum Computing Stack Exchange,” Accessed: Jul. 17, 2025. [Online]. Available: https://quantumcomputing. stackexchange.com/questions/29784/how-to-check-a-given-unitary- evolut...

    2025

  49. [59]

    Fault testing for reversible circuits,

    K. Patel, J. Hayes, and I. Markov, “Fault testing for reversible circuits,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 23, no. 8, pp. 1220–1230, 2004.DOI: 10.1109/TCAD. 2004.831576

    2004

  50. [60]

    Ping-pong test: Compact test vector generation for reversible circuits,

    M. Zamani, M. B. Tahoori, and K. Chakrabarty, “Ping-pong test: Compact test vector generation for reversible circuits,” in2012 IEEE 30th VLSI Test Symposium (VTS), 2012, pp. 164–169.DOI: 10.1109/ VTS.2012.6231097

    2012

  51. [61]

    A testing scheme for mixed-control based reversible circuits,

    B. Mondal, C. Bandyopadhyay, and H. Rahaman, “A testing scheme for mixed-control based reversible circuits,” in2016 Sixth International Symposium on Embedded Computing and System Design (ISED), 2016, pp. 96–100.DOI: 10.1109/ISED.2016.7977062

    2016

  52. [62]

    Fault testing for reversible circuits,

    K. Patel, J. Hayes, and I. Markov, “Fault testing for reversible circuits,” inProceedings. 21st VLSI Test Symposium, 2003., 2003, pp. 410–416. DOI: 10.1109/VTEST.2003.1197682

    2003 arXiv

  53. [63]

    Testing of reversible combinational circuits,

    Y . Syamala, A. V . N. Tilak, and K. Srilakshmi, “Testing of reversible combinational circuits,” inAdvances in Communication, Network, and Computing, V . V . Das and J. Stephen, Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 46–53,ISBN: 978-3-642-35615-5

    2012

  54. [64]

    A new online testing technique for reversible circuits,

    J. Mondal and D. K. Das, “A new online testing technique for reversible circuits,”IET Quantum Communication, vol. 3, no. 1, pp. 50–59, 2022. DOI: https://doi.org/10.1049/qtc2.12035

    2022 doi

  55. [65]

    Developers,Cirq, version v1.4.0, May 2024.DOI: 10.5281/zenodo

    C. Developers,Cirq, version v1.4.0, May 2024.DOI: 10.5281/zenodo. 11398048

    2024 doi

  56. [66]

    Bergholm, J

    V . Bergholm, J. Izaac, M. Schuld, C. Gogolin, S. Ahmed, V . Ajith, M. S. Alam, G. Alonso-Linaje, B AkashNarayanan, A. Asadi, et al.,Pennylane: Automatic differentiation of hybrid quantum-classical computations, 2018.DOI: 10.48550/arXiv.1811.04968 arXiv: 1811.04968

    2018 doi arXiv

  57. [67]

    Comparing algorithms for loading classical datasets into quantum memory,

    A. Miranskyy, M. Khan, and U. C. Mendes, “Comparing algorithms for loading classical datasets into quantum memory,” inIEEE International Conference on Quantum Computing and Engineering, IEEE, 2024, pp. 234–238.DOI: 10.1109/QCE60285.2024.10284

    2024

  58. [68]

    Mutation testing of quantum programs: A case study with qiskit,

    D. Fortunato, J. CAMPOS, and R. ABREU, “Mutation testing of quantum programs: A case study with qiskit,”IEEE Transactions on Quantum Engineering, vol. 3, pp. 1–17, 2022.DOI: 10.1109/TQE.2022. 3195061

    2022 doi

  59. [69]

    Muskit: A Mutation Analysis Tool for Quantum Software Testing,

    E. Mendiluze, S. Ali, P. Arcaini, and T. Yue, “Muskit: A Mutation Analysis Tool for Quantum Software Testing,” in2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021, pp. 1266–1270.DOI: 10.1109/ASE51524.2021.9678563

    2021

  60. [70]

    A Hitchhiker’s Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering,

    A. Arcuri and L. Briand, “A Hitchhiker’s Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering,”Software Testing, Verification and Reliability, vol. 24, no. 3, pp. 219–250, 2014, ISSN: 1099-1689.DOI: 10.1002/stvr.1486

    2014 doi

  61. [71]

    Yin,Case Study Research: Design and Methods(Applied Social Research Methods)

    R. Yin,Case Study Research: Design and Methods(Applied Social Research Methods). SAGE Publications, 2009,ISBN: 9781412960991

    2009

  62. [72]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. Ohlsson, B. Regnell, and A. Wessl´en,Experimentation in Software Engineering(Computer Science). Springer Berlin Heidelberg, 2012,ISBN: 9783642290442

    2012

  63. [73]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” inProceedings of the 33rd international conference on software engineering, 2011, pp. 1–10

    2011

  64. [74]

    Construct validity in software engineering,

    D. I. Sjøberg and G. R. Bergersen, “Construct validity in software engineering,”IEEE Transactions on Software Engineering, vol. 49, no. 3, pp. 1374–1396, 2022

    2022

  65. [75]

    Fundamental limits of quantum error mitigation,

    R. Takagi, S. Endo, S. Minagawa, and M. Gu, “Fundamental limits of quantum error mitigation,”npj Quantum Information, vol. 8, no. 1, p. 114, 2022. 18

    2022

  66. [76]

    Quantum error mitigation,

    Z. Cai, R. Babbush, S. C. Benjamin, S. Endo, W. J. Huggins, Y . Li, J. R. McClean, and T. E. O’Brien, “Quantum error mitigation,”Reviews of Modern Physics, vol. 95, no. 4, p. 045 005, 2023

    2023

  67. [77]

    IBM lays out clear path to fault-tolerant quantum computing

    R. Mandelbaum, J. Gambetta, J. Chow, T. Mittal, T. J. Yoder, A. Cross, and M. Steffen. “IBM lays out clear path to fault-tolerant quantum computing. ”[Online]. Available: https://www.ibm.com/quantum/blog/ large-scale-ftqc

  68. [78]

    Roads towards fault- tolerant universal quantum computation,

    E. T. Campbell, B. M. Terhal, and C. Vuillot, “Roads towards fault- tolerant universal quantum computation,”Nature, vol. 549, no. 7671, pp. 172–179, 2017.DOI: 10.1038/nature23460

    2017 doi

  69. [79]

    Six strategies for generalizing software engineering theories,

    R. J. Wieringa and M. Daneva, “Six strategies for generalizing software engineering theories,”Science of computer programming, vol. 101, pp. 136–152, 2015.DOI: 10.1016/J.SCICO.2014.11.013

    2015 doi

  70. [80]

    Mutation testing of quan- tum programs written in qiskit,

    D. Fortunato, J. Campos, and R. Abreu, “Mutation testing of quan- tum programs written in qiskit,” inProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ser. ICSE ’22, Pittsburgh, Pennsylvania: Association for Computing Mac...

    2022

  71. [81]

    QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit,

    D. Fortunato, J. Campos, and R. Abreu, “QMutPy: a mutation testing tool for Quantum algorithms and applications in Qiskit,” inProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2022, Virtual, South Korea: Association for Co...

    2022

  72. [82]

    QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments,

    J. Campos and A. Souto, “QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments,” in2021 IEEE/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE), ...

    2021

  73. [83]

    Bugs4Q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs,

    P. Zhao, Z. Miao, S. Lan, and J. Zhao, “Bugs4Q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs,”Journal of Systems and Software, vol. 205, p. 111 805, 2023,ISSN: 0164-1212.DOI: 10.1016/j.jss.2023.111805

    2023

  74. [84]

    Coppersmith,An approximate fourier transform useful in quantum factoring, 2002.DOI: 10.48550/arXiv.quant-ph/0201067 arXiv: quant- ph/0201067

    D. Coppersmith,An approximate fourier transform useful in quantum factoring, 2002.DOI: 10.48550/arXiv.quant-ph/0201067 arXiv: quant- ph/0201067

    2002 doi arXiv

  75. [85]

    QFT — Qiskit v.1.3.1,

    “QFT — Qiskit v.1.3.1,” Accessed: Dec. 26, 2024. [Online]. Available: https://docs.quantum.ibm.com/api/qiskit/qiskit.circuit.library.QFT

    2024

  76. [86]

    A fast quantum mechanical algorithm for database search,

    L. K. Grover, “A fast quantum mechanical algorithm for database search,” inProceedings of the twenty-eighth annual ACM symposium on Theory of computing, 1996, pp. 212–219

    1996

  77. [87]

    MorphQ: Metamorphic testing of the qiskit quantum computing platform,

    M. Paltenghi and M. Pradel, “MorphQ: Metamorphic testing of the qiskit quantum computing platform,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, 2023, pp. 2413– 2424

    2023

  78. [88]

    Fuzz4all: Universal fuzzing with large language models,

    C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  79. [2021]

    [Online]

    arXiv: 2007.07047 [cs.SE]. [Online]. Available: https://arxiv. org/abs/2007.07047

    2007 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.