{"id":"09668f1f-9b75-4986-803e-868e5d8173e9","arxiv_id":"2505.09059","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Applying mutation-based fault localization to Qiskit programs works well on artificial faults but is much less effective on real-world faults; quantum-specific mutations drive most of the useful signal.","lead":"This paper tests a debugging method that finds likely error lines in quantum programs by making small changes to the code and observing which changes turn failing tests into passing ones. It finds the method works well on artificially injected bugs but much less well on real-world bugs from open-source quantum projects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BugsAqua's mutation-injected faults are trivially reversible under the same QMutPy operators MBFL uses, so the RQ1 1.2% vs 19.4% gap may be an artifact of benchmark construction rather than real-world difficulty.","rationale":"The reader's weakest assumption was the representativeness of the 23 reproducible Bugs4Q bugs. That is a legitimate external-validity concern, but the more load-bearing internal-validity issue is the construction of the artificial benchmark: because BugsAqua faults are generated with the same mutation operators MBFL later uses, many are reversible by construction, making the artificial EXAM scores artificially low. This directly threatens the strongest claim, not just its generalizability. I still would not reject the paper: RQ2 and the SBFL comparison are informative, and the reversibility confound can be checked and reported. The appropriate outcome remains a conditional acceptance, with the authors required to either filter non-reversible BugsAqua faults or show the result is unchanged when doing so. I partially agree with the reader: their concern is real but different; the reversible-mutant confound is closer to the center of the argument.","tokens_in":9855,"tokens_out":7339,"duration_ms":75556,"concrete_test":"Reconstruct BugsAqua (or request the dataset from the authors), and for each of the 305 buggy programs check whether the original passing program, or any mutant generated from the faulty statement by the 23 QMutPy operators, makes the failing tests pass. Recompute the median worst-case EXAM restricted to the subset with no such reversible mutant. If the restricted median rises toward the 19.4% reported for Bugs4Q, RQ1's comparison is an artifact of reversible mutation; if it stays near 1.2%, the claim survives. Also report the share of BugsAqua cases that are reversible, since that quantifies the confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BugsAqua is constructed by applying QMutPy mutation operators to correct Qiskit-Aqua programs, and MBFL is then run by applying the same 23 operators to the buggy program (Section 3.3). For any injected fault that is a one-step replacement (for example, '+' mutated to '-' or quantum gate X mutated to Y), the original passing program is itself among the mutants of the faulty statement, because re-applying the replacement operator to the mutated statement can yield the original operation. Equation (1) then assigns that statement a maximum contribution: the failing test becomes passing in that mutant. The paper explicitly notes the existence of 'reversible' mutants in Section 4.1 as an explanation for the RQ1 gap, but it never controls for this property in the comparison. Since Bugs4Q contains real, mostly non-reversible faults while BugsAqua contains a large fraction of trivially reversible faults, the headline result that real-world faults are harder (median worst-case EXAM 19.4% vs 1.2%) may be substantially driven by the way the artificial benchmark was built, not by an intrinsic difference between real-world and artificial faults. The central claim therefore has an uncontrolled confound at its core.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies mutation-based fault localization (MBFL) to quantum programs written in Qiskit, using the QMutPy tool to generate mutants of buggy programs and the MUSE-style suspiciousness formula of Eq. (1) to rank statements. The evaluation uses 23 real-world faults from Bugs4Q and 305 artificial faults from a newly constructed benchmark, BugsAqua, which is built by injecting QMutPy mutants into Qiskit-Aqua programs. The central results are: (RQ1) real-world faults are harder for MBFL than artificial faults, with median worst-case EXAM scores of 19.4% versus 1.2%; (RQ2) quantum mutation operations produce more behavior-changing mutants per operator than classical operations; and a comparison showing MBFL outperforms SBFL in the worst-case scenario on BugsAqua. The authors conclude that MBFL shows promise for quantum programs but that existing mutation operations are limited for realistic faults, and they suggest enriching the set of quantum mutation operations.","tokens_in":10032,"tokens_out":3740,"duration_ms":38239,"significance":"If the findings hold, this is a useful contribution to the emerging area of quantum software testing and debugging. The paper is, to my knowledge, among the first to compare MBFL performance on real-world versus artificially injected quantum program faults, and it directly follows the methodology of Pearson et al. for classical programs. The use of publicly available benchmarks (Bugs4Q, QMutPy) and the explicit discussion of failure cases (e.g., Listing 2) are strengths. The paper also provides a falsifiable ranking result: quantum mutation operations dominate behavior-changing mutants for Bugs4Q, which is a concrete and checkable claim. However, the current support for the headline claim is bounded by methodological gaps: the instrumentation used to compute f_P(s) is not described, the artificial benchmark is constructed with the same mutation operators that MBFL uses for scoring, and a large fraction of both real and artificial faults is excluded. These issues do not automatically invalidate the direction of the result, but they currently prevent the paper from supporting its strong comparative conclusion.","major_comments":[{"comment":"The suspiciousness formula relies on f_P(s), defined as the set of failed test cases when statement s is executed in program P. The paper never describes how statement execution is tracked for quantum programs, which contain both classical Python control flow and quantum operations such as gates and measurements. Is the instrumentation done at the AST level, through Qiskit circuit inspection, or via some other mechanism? Without this description, the experimental procedure is not reproducible and the reader cannot assess whether statements that are not executed on some paths are handled correctly.","section":"Section 3.3, Eq. (1)"},{"comment":"The BugsAqua benchmark is constructed by applying the same 23 QMutPy mutation operators that MBFL later uses to score statements. As the authors note in Section 4.1, many of these injected faults are 'reversible': a one-step replacement can restore the original passing program, which makes the faulty statement trivially suspicious in Eq. (1). The paper reports this as an explanation for the RQ1 gap, but it does not control for it. Since the headline result (median worst-case EXAM 19.4% vs. 1.2%) compares an artificial benchmark that is partly easy by construction against a real-world benchmark that is not, the comparison is confounded. I ask the authors to (a) quantify the proportion of BugsAqua faults that are reversible under the 23 operators, (b) repeat the RQ1 comparison on the subset of non-reversible artificial faults, or otherwise show that the conclusion is not an artifact of benchmark construction, and (c) clearly separate the 'intrinsic difficulty' claim from the 'benchmark construction' claim.","section":"Section 3.2.2 and Section 4.1 (RQ1)"},{"comment":"The paper excludes 289 of 594 BugsAqua mutants solely because their execution time exceeds one hour, and it uses the remaining 305 as the artificial benchmark. This is a selection on execution time, which can correlate with program size, loop structure, or the number of quantum operations. The paper does not compare the included and excluded mutants in terms of mutation operator type, program size, or the resulting EXAM scores. A sensitivity analysis is needed to show that the median EXAM results are stable under this exclusion. At a minimum, report the distribution of execution times and the characteristics of the excluded mutants.","section":"Section 3.2.2 and Section 4.1"},{"comment":"RQ2 is answered only by counting behavior-changing mutants (Table 1), but the paper's contribution statement says 'quantum mutation operations are effective for MBFL.' Behavior-changing mutants are a necessary input to Eq. (1), but they do not directly imply lower EXAM scores; a mutation operator could produce many behavior-changing mutants that all localize to the wrong statement. To support the effectiveness claim, the authors should report EXAM scores broken down by mutation-operation type, or at least show that the statements mutated by quantum operators are more often the faulty ones. As written, RQ2 is about mutation strength, not fault-localization effectiveness.","section":"Section 4.2 and Section 1 (Contribution 2)"},{"comment":"The real-world evaluation relies on 23 of the 42 Bugs4Q bugs, with the other 19 excluded because they could not be reproduced. The paper acknowledges this, but it does not assess whether the excluded bugs differ systematically from the included ones (e.g., in fault type, quantum versus classical root cause, or program size). Given that the central RQ1 conclusion is about real-world faults, the representativeness of the 23-bug subset is load-bearing. I request an analysis of the excluded cases from the Bugs4Q replication package, or a clear argument that the non-reproduction is orthogonal to the fault-localization behavior.","section":"Section 3.2.1 and Section 4.1"}],"minor_comments":[{"comment":"The abstract reports '305 artificially induced faults,' but Section 3.2.2 states that QMutPy generated 2,361 mutants and 594 had at least one failing test, after which the timeout exclusion left 305. The wording in the abstract should distinguish faults from mutants and mention the filtering step.","section":"Abstract and Section 3.2.2"},{"comment":"There is a typo in the Answer to RQ1: 'Additionaly' should be 'Additionally.'","section":"Section 4.1"},{"comment":"The phrase 'a level that satisfies approximately 10% of practitioners' with reference [12] is unclear. Reference [12] is about practitioners' expectations on fault localization, not about execution timeout thresholds; please clarify how the one-hour timeout was derived from that source.","section":"Section 3.2.2"},{"comment":"Equation (1) does not define the behavior when f_P(s) is empty, which can occur for statements not executed by any failing test. It would be helpful to state the convention used in that case.","section":"Section 3.4"},{"comment":"The statement that SBFL 'cannot be applied' to Bugs4Q because there is only one failing test case is too strong. Ochiai and Tarantula can still be computed with a single failing test and multiple passing tests; the issue is that the resulting scores may be degenerate. Please rephrase to say that SBFL is not meaningful or not reliably applicable in this setting.","section":"Section 5.2"},{"comment":"The 'Sig. level' column uses '**' and '-' without a legend in the table caption. Please add a footnote explaining that '-' means p > 0.05.","section":"Table 2"},{"comment":"The threats section covers many of the concerns raised above, but it does not mention the circularity/reversibility issue in the construction of BugsAqua. Since the paper explicitly identifies reversible mutants as the explanation for RQ1, this threat should be acknowledged here as well.","section":"Section 6 (Threats to Validity)"}],"recommendation":"major_revision","confidential_remarks":"The central result is plausible and related work (Pearson et al.) shows a similar pattern for Java programs, so I do not recommend rejection. However, the reversibility confound is genuinely load-bearing: if a large fraction of BugsAqua faults are reversible under the 23 QMutPy operators, then the 1.2% median EXAM is partly an artifact of using the same mutation operators for injection and for localization. The required additional analyses (reversibility quantification, non-reversible subset comparison, timeout-exclusion sensitivity) are feasible with the existing infrastructure and should be added. The missing instrumentation description is also a reproducibility blocker that must be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is the first empirical study of mutation-based fault localization for quantum programs, and it is a reasonably honest one. The headline gap between real and artificial faults is real but inflated because the artificial benchmark is generated with the same mutation operators MBFL uses.\n\nThe paper does something new: it applies the Pearson et al. methodology to Qiskit programs, using Bugs4Q real bugs and a new BugsAqua artificial set. RQ2 is the more interesting result — quantum mutation operations dominate behavior-changing mutants for real bugs (93.9%), which suggests quantum-specific operators are essential for localization in this domain. The authors also compare MBFL with SBFL on the artificial benchmark and find MBFL more stable in the worst case. They openly discuss the reversible-mutant explanation and list threats to validity. That honesty is commendable.\n\nThe soft spots are substantial but mostly about benchmark construction. 19 of 42 Bugs4Q bugs are excluded because of environment incompatibilities; that could bias the real-world sample. For BugsAqua, 289 of 594 mutants time out and are dropped, removing the hardest cases. More importantly, the stress-test concern is real: BugsAqua faults are created by the same QMutPy operators used by MBFL, so many are one-step reversible. Equation (1) rewards exactly those reversals, so the 1.2% median EXAM is partly a circular result. The paper acknowledges this but does not control for it, so the RQ1 comparison is not as clean as it looks.\n\nThe real-world result still stands: median worst-case EXAM of 19.4% with about 40% of cases at 100% means MBFL is not reliable for real bugs. That is a useful, falsifiable finding. But the paper's central claim about the gap needs qualification.\n\nAlso missing: no artifact, no description of statement-coverage instrumentation, and the SBFL comparison only works on the artificial benchmark because Bugs4Q has one test per bug. Those are minor-to-moderate issues.\n\nWho this is for: quantum software engineering researchers and anyone building fault localization tools for quantum programs. It deserves a serious referee because it opens a new evaluation area, but it needs revisions to control for the reversible-mutant confound and to release the benchmark.\n\nMy recommendation: engage with it, but treat the artificial-vs-real comparison as suggestive, not definitive.","headline":"First MBFL-for-quantum study with a real finding, but the artificial benchmark's construction inflates the headline gap.","tokens_in":10644,"tokens_out":2369,"would_cite":false,"duration_ms":22925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mutation-based fault localization works well on artificial quantum bugs but poorly on real-world ones.","keywords":["quantum program fault localization","mutation-based fault localization","mutation testing","quantum mutation operations","Qiskit","EXAM score","spectrum-based fault localization","real-world fault benchmarks"],"falsifier":"Run the same MBFL pipeline on all 42 Bugs4Q bugs with dependency versions pinned to the benchmark's original setup, and compare the median worst-case EXAM score. If it falls well below 19.4%, the reported real-versus-artificial gap is an artifact of the 23-bug reproducible subset; if adding a missing-initialization mutation operation drops the median into single digits, the paper's proposed direction is confirmed.","tokens_in":9597,"feed_emoji":"🐛","tokens_out":7692,"duration_ms":66471,"temperature":0.7,"pith_summary":"This paper applies mutation-based fault localization (MBFL) to quantum programs written in Qiskit and tests it on 23 real-world bugs and 305 artificially injected bugs. It finds that artificial bugs are much easier: the median EXAM score (percentage of code examined before hitting the faulty statement, lower is better) is 1.2% in the worst case for artificial bugs versus 19.4% for real-world bugs. The gap matters because any evaluation built only on injected faults will overstate how well MBFL would work in practice. The paper also shows that quantum-specific mutation operations, not classical ones, produce nearly all the behavior-changing mutants behind successful localization on real bugs.","feed_headline":"Real quantum bugs resist mutation-based debugging","feed_subtitle":"MBFL scans 1.2% of code for injected faults, 19.4% for real ones","key_machinery":"The load-bearing mechanism is the MUSE-style suspiciousness formula, which scores each statement $s$ by the fraction of failed tests of the original program that are flipped to passing in mutants of $s$: $S(s) = \\frac{1}{|mut(s)|}\\sum_{m\\in mut(s)} |f_P(s)\\cap p_m|/|f_P(s)|$. Localization quality is then read off the EXAM score, defined as the percentage of ranked statements a developer must inspect before reaching the first faulty statement, with best- and worst-case tie handling. Mutants come from QMutPy, configured with 20 classical and 3 usable quantum mutation operations (gate deletion, gate replacement, and measurement deletion); the study measures how many of these mutants are behavior-changing, since only those can change the suspiciousness scores.","core_discovery":"The central finding is that mutation-based fault localization transfers to quantum programs but its effectiveness collapses when faults are real. On a curated subset of 23 reproducible bugs from the Bugs4Q benchmark, the worst-case median EXAM score is 19.4%, and roughly 40% of those bugs get a worst-case EXAM of 100% — the faulty statement is no better than last in the ranking. On 305 artificially injected faults in Qiskit-Aqua programs, the same pipeline reaches a worst-case median EXAM of 1.2%. The paper attributes the gap to the expressiveness of the mutation operations: real faults such as a forgotten qubit-initialization pattern are not reversible or reachable by single-statement gate addition, deletion, or replacement, while artificial faults often are. A secondary result is that quantum mutation operations outperform classical ones in producing behavior-changing mutants, and that MBFL beats spectrum-based fault localization on the artificial benchmark in the worst-case tie scenario, while spectrum-based methods cannot even be run on Bugs4Q because each bug has only one failing test.","pith_inferences":["If the 19 excluded Bugs4Q bugs differ systematically from the 23 reproducible ones, the true gap between artificial and real-world difficulty could be larger or smaller than the reported 19.4% versus 1.2% median; the paper's validity section acknowledges this bias but does not bound it.","Adding a missing-initialization mutation operation and re-running the same pipeline on Bugs4Q would give a direct test of whether operator coverage, rather than test quality or fault complexity, explains the poor real-world scores.","The single-failing-test advantage of MBFL over SBFL may extend beyond quantum software to any domain where bug reports arrive with a single reproduction test."],"forward_implications":["Evaluations of MBFL for quantum programs must include real-world faults; benchmarks built only from injected mutants overstate practical effectiveness.","Quantum mutation operations should be the focus of MBFL tooling for quantum programs, since they produced 93.9% of behavior-changing mutants on real bugs despite being 3 of 23 operation types.","When a bug has a single failing test, spectrum-based fault localization cannot rank statements at all, while MBFL still produces a suspiciousness ordering.","Enriching mutation operations with recurring quantum patterns, such as a missing qubit-initialization operation, is a concrete next step implied by the failure cases.","On the artificial benchmark, MBFL resolves suspiciousness ties more conservatively than SBFL, yielding a significantly better worst-case EXAM (p<0.01, large effect size)."],"supporting_citations":[{"why":"Supplies the real-world bug benchmark: 42 buggy Qiskit programs with fixed versions and manually written tests.","marker":"[22]"},{"why":"Provides QMutPy, the mutation tool whose quantum and classical mutation operations generate the mutants used in MBFL.","marker":"[6]"},{"why":"The classical Java study whose artificial-versus-real-world comparison and EXAM metric this work extends to quantum programs.","marker":"[16]"},{"why":"Defines MUSE, the mutation-based fault localization technique whose suspiciousness formula the paper simplifies.","marker":"[14]"},{"why":"Defines the EXAM score and the best-case/worst-case tie-handling adjustment used for evaluation.","marker":"[21]"},{"why":"The Ochiai spectrum-based formula used as an SBFL baseline for comparison.","marker":"[1]"},{"why":"The Tarantula spectrum-based formula used as the second SBFL baseline.","marker":"[9]"},{"why":"An alternative cost-based fault localization approach for quantum programs, contrasted for limited applicability to mixed classical/quantum code.","marker":"[18]"}],"fun_headline_variants":["Mutation debugging: 1.2% vs 19.4% code scan for quantum bugs","Quantum bug locator stumbles on real faults","Real quantum bugs defy mutation-based search","Quantum debugging gap: 16x harder for real bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 23 Bugs4Q bugs that reproduce in the authors' environment, each paired with a single manually written test, represent real-world quantum program faults well enough for the poor EXAM scores to be a true property of the technique rather than of the subset.","fun_headline_variants_meta":{"raw":{"variants":["Mutation debugging: 1.2% vs 19.4% code scan for quantum bugs","Quantum bug locator stumbles on real faults","Real quantum bugs defy mutation-based search","Quantum debugging gap: 16x harder for real bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2409,"prompt_tokens":958,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1382}},"tokens_in":574,"tokens_out":1451,"duration_ms":9207,"temperature":1.0,"reasoning_tokens":1382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:40:54.240674+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MBFL pipeline on all 42 Bugs4Q bugs with dependency versions pinned to the benchmark's original setup, and compare the median worst-case EXAM score. If it falls well below 19.4%, the reported real-versus-artificial gap is an artifact of the 23-bug reproducible subset; if adding a missing-initialization mutation operation drops the median into single digits, the paper's proposed direction is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the real-world bug benchmark: 42 buggy Qiskit programs with fixed versions and manually written tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides QMutPy, the mutation tool whose quantum and classical mutation operations generate the mutants used in MBFL."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The classical Java study whose artificial-versus-real-world comparison and EXAM metric this work extends to quantum programs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MUSE, the mutation-based fault localization technique whose suspiciousness formula the paper simplifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Tarantula spectrum-based formula used as the second SBFL baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An alternative cost-based fault localization approach for quantum programs, contrasted for limited applicability to mixed classical/quantum code."}],"review_version":1}