Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Only 24% of quantum-computing papers share code; 64% of shared code fails to run in a clean environment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:22 UTC pith:JM3R6BQA

load-bearing objection Useful first measurement of QC reproducibility, but the headline code-availability rate is not as clean as it looks — the manual estimate inherits the automated keyword scan's recall, and the executability denominator is ambiguous. the 4 major comments →

arxiv 2607.08348 v2 pith:JM3R6BQA submitted 2026-07-09 quant-ph

Works on My QPU: Reproducibility in Quantum Computing Research

Dominik K\"oster , Maja Franz , Benjamin Zec , Nicole Hoess , Ralf Ramsauer , Wolfgang Mauerer This is my paper
classification quant-ph
keywords reproducibilityquantum computingresearch artefactsNISQsoftware environmentscode availabilitydependenciesempirical study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that reproducibility is the exception, not the rule, in quantum computing research. A manually reviewed sample of 127 papers found that only 24.4% provide code artefacts, and among those, 64.5% fail to execute in a freshly provisioned environment. A large automated scan of nearly 5000 papers found a consistent code-availability rate of 26.8%. The authors argue that the main cause is not simply aging dependencies, but incomplete or implicit environment specifications that the original developers never recorded. They offer practical recommendations, centered on explicit, machine-readable, pinned environments, and provide their own reproduction package as a blueprint.

Core claim

Using a five-question framework covering code availability, environment specification, documentation, hardware specification, and executability, the authors find that only about one quarter of recent quantum-computing papers provide accessible code, and that most of that code does not run as documented in a clean environment. The manual study of 127 NISQ-era papers gives a code-availability rate of 24.4%, with 64.5% of those artefacts failing to execute; the automated screening of 4966 papers yields a corroborating 26.8% availability. Among papers with code, more than half lack a complete machine-readable environment specification, and most failures trace to implicit assumptions such as unpi

What carries the argument

The central instrument is a five-question reproducibility framework (RQ1 code availability, RQ2 environment specification, RQ3 build-and-run documentation, RQ4 hardware specification, RQ5 executability) applied through two complementary pipelines: a manual, human-validated analysis of a filtered 127-paper sample, and an automated full-text and repository scan of the full 4966-paper corpus. The framework lets the authors convert a qualitative idea—'can this paper be reproduced?'—into measurable indicators, and the parallel use of manual and automated methods lets them check whether sampling bias distorts the headline rates.

Load-bearing premise

The headline rates rest on the assumption that restricting the manual sample to NISQ-titled papers with DOIs found in both databases, and using keyword hits plus accessible repository links as proxies for code availability, yields a representative cross-section of quantum-computing research.

What would settle it

Take a fresh, independently drawn random sample of, say, 30 NISQ-era quantum-computing papers that provide code, and attempt reproduction in a clean environment following only the papers' documentation. If the success rate substantially exceeds the reported 35.5%, the central executability claim is too pessimistic; if a broader random sample of quantum papers without the NISQ filter shows code availability far above 25%, the representativeness of the sampling is in doubt.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the rates hold, most published quantum-computing results cannot be independently verified by executing the authors' code, since only about one in four papers shares artefacts and most of those do not run.
  • The presence of code, documentation, and environment files is not a reliable proxy for reproducibility; actual execution in a clean environment is necessary.
  • Common failure modes are concrete: unpinned or missing dependencies, unspecified language versions, hardcoded local paths, API version conflicts, `latest` container tags, and incomplete instructions.
  • Authors should provide pinned, machine-readable environment definitions, a single entry point, validation in independent environments, and simulator-based fallbacks plus raw data when hardware access is temporary.
  • The paper's own reproduction package, containing containerised and source-rebuildable environments, demonstrates how the recommendations can be operationalised.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 64.5% failure rate is representative, a broader re-execution campaign of previously published quantum experiments could uncover quantitative differences in reported results caused by toolchain drift—an extension the authors hint at but do not perform.
  • The five-question framework could be transferred directly to other fast-moving software-dependent fields, such as machine learning or scientific computing, where similar implicit-environment problems are likely.
  • The automated screening's reliance on keyword hits and accessible repository links may undercount artefacts stored in supplements, institutional repositories, or non-standard hosts, so true code availability could be somewhat higher than the reported 24–27%.
  • Future community adoption may hinge less on a specific tool like Nix or Guix and more on a lightweight, journal- or conference-enforced checklist that requires a clean-environment run before acceptance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a two-part empirical study of reproducibility in quantum computing research. The authors construct a corpus of 4,966 QC papers (2021–2026, arXiv quant-ph, keyword-filtered), derive a manually vetted NISQ-focused subsample of 127 papers, and apply a five-question framework (RQ1–RQ5: code availability, environment specification, documentation, hardware specification, executability). They report that 24.4% of the manual sample provide code artefacts, with 64.5% of those failing to execute in a clean environment. An automated large-scale scan of the 4,966-paper corpus reports a code-availability rate of 26.8%, which the authors read as corroboration. The paper closes with recommendations centered on declarative, machine-readable environment specifications (Nix/devenv) and provides its own reproduction package as a template.

Significance. If the headline estimates are accurate, the paper provides useful quantitative evidence that reproducibility artefacts in quantum computing are both scarce and frequently non-executable, extending a well-known problem from classical software research into a domain with rapidly evolving toolchains. The study's strengths include a clearly described sampling pipeline, a reproducible assessment pipeline (GitHub/Zenodo links), a five-question framework that is simple to reuse, a combined manual/automated design, and an explicit reproduction package illustrating the recommendations. The consistency between the manual (24.4%) and automated (26.8%) code-availability rates gives some confidence, though the two pipelines share detection heuristics. The main value is as a community survey and a practical call for more complete environment specifications; the technical contribution is moderate but appropriate for a journal in reproducibility/software engineering.

major comments (4)
  1. [§II-C-5, Table II, RQ5 denominator] The claimed 'manual' code-availability rate (24.4%) and the automated rate (26.8%) are not independent: both rely on the same keyword-based scan for repositories (github/gitlab/bitbucket/zenodo, 'code availab(le|ility)', etc.). In the manual pipeline, the 61 of 127 papers that did not match the keyword scan were never human-checked, so the manual rate inherits any false negatives of the automated scan. The paper should report precision/recall of the keyword scan against a manually annotated gold standard (e.g., the 127-paper set, or a random subset thereof) and bound the bias. Without this, the 'corroboration' between 24.4% and 26.8% is weaker than stated.
  2. [§II-B-1, paper sampling] RQ5 (Executability) is reported as 35.5% executable / 64.5% failing, but the denominator is ambiguous. Table II shows Executability with 'Yes 8.7%' over all 127 (11/127 = 8.7%), while §II-C-5 says 35.5% of examined papers with code availability. The reader must infer that the 64.5% failure rate is conditional on 31 code-available papers, but the text should state this explicitly and reconcile the 'N/A' 7.9% (code on request). The 35.5% figure also lacks a confidence interval; with n≈31, the precision is low and should be reported.
  3. [§II-B-3, automated pipeline] The manual sample is narrowly filtered: papers must contain 'NISQ' in title/abstract, have a DOI, appear in both arXiv and Semantic Scholar, and contain experimental keywords. This may produce a sample that is not representative of the full QC landscape; because the same filters do not apply to the automated corpus, the similarity in code-availability rates could be coincidental or driven by the shared keyword scan rather than by population similarity. The paper should provide a sensitivity analysis (e.g., code-availability rate among the 648 NISQ papers before the DOI/database filters, or among random non-NISQ papers) to show the 24.4% is not an artifact of the narrow filters.
  4. [Abstract] The automated large-scale analysis treats HTTP accessibility plus the presence of environment files (requirements.txt, Dockerfile) and README as indicators of reproducibility readiness. This is a reasonable proxy, but the paper's own manual RQ5 results show that these signs are not predictive of executability. The claim that the automated analysis 'corroborates' the manual findings for RQ2–RQ4 is thus limited to availability, not quality. The paper should state this limitation explicitly and avoid implying that 26.8% is the same kind of rate as 24.4%.
minor comments (5)
  1. [Abstract] The phrase 'combined manual and automated large-scale analysis' could be clearer: the manual and automated pipelines are separate, with the manual pipeline itself using an automated pre-screen. Also, 'only 24.4%' is stated without the margin of error; a small n=127 sample warrants a confidence interval.
  2. [§II-C-1] The sentence 'This aligns closely with the large-scale analysis' should acknowledge that the 26.8% figure uses a different operationalization (accessible repository links) than the manual 24.4% (code artefacts after manual verification). A table column defining each metric would help.
  3. [Table II] Table II is compact but the 'Partially' and 'N/A' rows are hard to parse. For example, Hardware Spec 'N/A' differs between manual (26.8%) and automated (8.9%) and should be footnoted to explain what N/A means in each pipeline (e.g., simulation-only papers vs. no hardware keywords).
  4. [§II-B-2-a] The 'full-text search' for experimental keywords (experiments, experimental result, numerical) is described in one sentence; the exact boolean logic and whether these keywords were searched in the same PDF full text as the later code scan should be clarified.
  5. [§IV] The recommendations are reasonable but mostly generic best practices. The Nix/devenv discussion is valuable, but the paper could cite additional QC-specific reproducibility efforts, e.g., reproducible builds for QC (Ref. [17]) is mentioned only in passing.

Circularity Check

0 steps flagged

No significant circularity: the claimed reproducibility rates are empirical measurements on external paper corpora, not derived from fitted parameters or the paper's own assumptions.

full rationale

The central quantitative claims—24.4% manual code availability, 64.5% manual non-executability, and 26.8% automated code availability—are outputs of a measurement pipeline applied to externally collected paper corpora. They are not derived from fitted parameters, theoretical identities, or the paper's own recommendations. The manual and automated pipelines do share the same keyword-based full-text screening, so the agreement between 24.4% and 26.8% is not fully independent corroboration; this is a sampling/measurement limitation, not a circular reduction, because no quantity is defined in terms of the outcome it is used to estimate and no equation or fitted value is renamed as a prediction. Self-citations ([2], [10], [37]) appear in background, motivation, and discussion of environment tooling, but none is load-bearing for the measured rates: the empirical results would stand unchanged if those citations were removed. No uniqueness theorem, ansatz, or prior result by the same authors is used to force the conclusions. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No fitted mathematical parameters; the central claim rests on hand-chosen sampling and screening choices (listed) and on the domain assumption that software-artifact availability proxies reproducibility. No new entities are introduced.

free parameters (4)
  • Manual sample filters = NISQ in title/abstract; DOI present; found in both arXiv and Semantic Scholar; contains 'experiments'/'experimental resu
    These hand-chosen filters define the 127-paper sample from which code availability and executability percentages are computed; changing them changes headline rates.
  • Automated keyword indicator set = git(hub|lab|ee), bitbucket, zenodo, source-code, repository, code availab(le|ility), data availab(le|ility), download co
    The large-scale 26.8% code-availability rate is determined by this regex set; it is a hand-authored proxy, not a validated classifier.
  • Date range = 2021-01-01 to 2026-03-27
    Restricts both corpora; reproducibility rates may differ outside this window.
  • RQ2/RQ3 thresholds for executable testing = papers satisfying both environment specification and documentation
    Affects the RQ5 executability denominator; the text is ambiguous about whether all 31 code-bearing papers or only the RQ2+RQ3 subset were execution-tested.
axioms (4)
  • domain assumption A five-question framework (code availability, environment, documentation, hardware, executability) captures reproducibility
    Introduced in Table I; the entire measurement is defined through this rubric, which emphasizes software artifacts over experiment replication.
  • domain assumption Quantum-computing research papers can be identified by title/abstract terms 'quantum computing' plus 'algorithm' or 'software', excluding surveys/reviews
    Sampling in §II-B-1; corpus composition depends on this query.
  • domain assumption NISQ-era papers (title/abstract contains 'NISQ') are representative of the QC research landscape
    Manual analysis is restricted to 648 NISQ papers and then to 127; generalization to all QC rests on this.
  • domain assumption PDF full-text keyword hits and accessible repository URLs are valid proxies for code availability
    Automated screening §II-B-3; false positives and false negatives are not quantified.

pith-pipeline@v1.3.0-alltime-deepseek · 9368 in / 13203 out tokens · 118517 ms · 2026-08-04T04:22:31.666530+00:00 · methodology

0 comments
read the original abstract

Quantum computing research increasingly depends on complex software stacks, yet the reproducibility of published results does not receive the priority and longevity mandated by recommendations of large international scientific bodies and best practices in software-centric systems research. In this paper, we present a combined manual and automated large-scale analysis of the reproducibility landscape in quantum computing research, quantify shortcomings, and derive actionable steps forward. We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability. To place these findings in a broader context, we conduct an automated large-scale screening of nearly 5000 quantum computing papers for the same reproducibility indicators. Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment. This assessment is corroborated by a large-scale automated analysis that yields a consistent code availability rate of 26.8%. Further, it shows that approximately one-third of the papers with accessible code lack machine-readable environment specifications. The results in this paper indicate that reproducibility is not yet consistently achieved in quantum computing research. In response, we outline a set of practical recommendations that address the observed failure modes and illustrate how reproducibility can be improved in practice.

Figures

Figures reproduced from arXiv: 2607.08348 by Benjamin Zec, Dominik K\"oster, Maja Franz, Nicole Hoess, Ralf Ramsauer, Wolfgang Mauerer.

Figure 1
Figure 1. Figure 1: Papers are retrieved via a structured query from arXiv and Semantic Scholar, merged and deduplicated. Subsequently [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 4 linked inside Pith

  1. [1]

    J. M. Gonz ´alez-Barahona and G. Robles, On the re- producibility of empirical software engineering studies based on data retrieved from development repositories, EMSE, 2012

    2012

  2. [2]

    Mauerer, S

    W. Mauerer, S. Klessinger, and S. Scherzinger, Beyond the badge: Reproducibility engineering as a lifetime skill, inProc. SEENG, 2022

    2022

  3. [3]

    Trautsch, S

    F. Trautsch, S. Herbold, P. Makedonski, et al., Ad- dressing problems with replicability and validity of repository mining studies through a smart data platform, EMSE, 2018

    2018

  4. [4]

    J. M. Gonzalez-Barahona and G. Robles, Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repos- itories,IST, 2023

    2023

  5. [5]

    J. T. Liang, C. Badea, C. Bird, et al., Can GPT-4 Replicate Empirical Software Engineering Research? PACMSE, 2024

    2024

  6. [6]

    T. Yue, W. Mauerer, S. Ali, et al., Challenges and oppor- tunities in quantum software architecture, inSoftware Architecture: Research Roadmaps from the Community, 2023

    2023

  7. [7]

    Senapati, Z

    P. Senapati, Z. Wang, W. Jiang, et al., Towards Redefin- ing the Reproducibility in Quantum Computing: A Data Analysis Approach on NISQ Devices, inProc. QCE, 2023

    2023

  8. [8]

    Carbonelli, M

    C. Carbonelli, M. Felderer, M. Jung, et al., Challenges for quantum software engineering: An industrial ap- plication scenario perspective, inQuantum Software: Aspects of Theory and System Design, 2024

    2024

  9. [9]

    Munasinghe, K

    T. Munasinghe, K. A. Cornell, J. Hendler, et al., A Knowledge-Based System for Managing Hardware De- pendency and Reproducibility in Quantum Machine Learning Workflows, inProc. IEEE BigData, 2025

    2025

  10. [10]

    Mauerer and S

    W. Mauerer and S. Scherzinger, 1-2-3 Reproducibility for Quantum Software Experiments, inProc. SANER, 2022

    2022

  11. [11]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Repro- ducibility of Build Environments through Space and Time, inProc. ICSE-NIER, 2024

    2024

  12. [12]

    Ramsauer and W

    R. Ramsauer and W. Mauerer, Towards system-level quantum-accelerator integration, inProc. QCE, 2025

    2025

  13. [13]

    Younis and C

    E. Younis and C. Iancu, Quantum Circuit Optimization and Transpilation via Parameterized Circuit Instantia- tion, inProc. QCE, 2022

    2022

  14. [14]

    Greiwe, T

    F. Greiwe, T. Kr ¨uger, and W. Mauerer, Effects of imper- fections on quantum algorithms: A software engineering perspective, inProc. QSW, 2023

    2023

  15. [15]

    Kr ¨uger and W

    T. Kr ¨uger and W. Mauerer, Out of the Loop: Structural Approximation of Optimisation Landscapes and non- Iterative Quantum Optimisation,Quantum, 2025

    2025

  16. [16]

    Schmidbauer, E

    L. Schmidbauer, E. Lobe, I. Schaefer, et al., It’s quick to be square: Fast quadratisation for quantum toolchains, TQC, 2026

    2026

  17. [17]

    I. M. Veiga and E. H ¨anggi, Reproducible builds for quantum computing, arXiv:2510.02251, 2025

    arXiv 2025

  18. [18]

    Y . Shi, P. Gokhale, P. Murali, et al., Resource-Efficient Quantum Computing by Breaking Abstractions,Proc. IEEE, 2020

    2020

  19. [19]

    Dasgupta and T

    S. Dasgupta and T. Humble, Impact of Unreliable Devices on Stability of Quantum Computations,TQC, 2024

    2024

  20. [20]

    Thelen, H

    S. Thelen, H. Safi, and W. Mauerer, Approximating under the influence of quantum noise and compute power, inProc. QCE, 2024

    2024

  21. [21]

    S. R. Maschek, J. Schwittalla, M. Franz, et al., Make some noise! measuring noise model quality in real- world quantum software, inProc. QSW, 2025

    2025

  22. [22]

    Javadi-Abhari, M

    A. Javadi-Abhari, M. Treinish, K. Krsulich, et al., Quan- tum computing with qiskit, arXiv:2405.08810, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Bergholm, J

    V . Bergholm, J. Izaac, M. Schuld, et al., Pennylane: Automatic differentiation of hybrid quantum-classical computations, arXiv:1811.04968, 2022

    Pith/arXiv arXiv 2022

  24. [24]

    Cirq Developers, Cirq, 2025

    2025

  25. [25]

    R. S. Smith, M. J. Curtis, and W. J. Zeng, A practical quantum instruction set architecture, arXiv:1608.03355, 2017

    Pith/arXiv arXiv 2017

  26. [26]

    Schulz, M

    M. Schulz, M. Ruefenacht, D. Kranzlm ¨uller, et al., Accelerating HPC With Quantum Computing: It Is a Software Challenge Too,CiSE, 2022

    2022

  27. [27]

    Zhong, S

    Z. Zhong, S. He, H. Wang, et al., An Empirical Study on Package-Level Deprecation in Python Ecosystem, in Proc. ICSE, 2025

    2025

  28. [28]

    D. B. Bose, T. Chan, M. Trimble, et al., AutoPyDep: A Recommendation System for Python Dependency Management Utilizing Graph-Based Analytics, inProc. FSE Companion, 2025

    2025

  29. [29]

    Fern ´andez-Osuna, R

    M. Fern ´andez-Osuna, R. P ´erez-Castillo, J. A. Cruz- Lemus, et al., Exploring design patterns in quantum software: A case study,Computing, 2025

    2025

  30. [30]

    L. J. Kitt and M. B. Cohen, MorphQ++: A Repro- ducibility Study of Metamorphic Testing on Quantum Compilers, inProc. ASE Workshops, 2024

    2024

  31. [31]

    Cardinal, I

    J. Cardinal, I. Benzarti, G. E. boussaidi, et al., Migrating QAOA from Qiskit 1.x to 2.x: An experience report, arXiv:2512.08245, 2026

    arXiv 2026

  32. [32]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Does Functional Package Management Enable Reproducible Builds at Scale? Yes. InProc. MSR, 2025

    2025

  33. [33]

    Devresse, F

    A. Devresse, F. Delalondre, and F. Sch ¨urmann, Nix based fully automated workflows and ecosystem to guarantee scientific result reproducibility across soft- ware environments and systems, inProc. SE-HPCCSE, 2015

    2015

  34. [34]

    Open Archives Initiative, arXiv API, 2026

    2026

  35. [35]

    Kinney, C

    R. Kinney, C. Anastasiades, R. Authur, et al., The semantic scholar open data platform, arXiv:2301.10140, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Docker does not guarantee reproducibility, arXiv:2601.12811, 2026

    arXiv 2026

  37. [37]

    Sch ¨onberger, M

    M. Sch ¨onberger, M. Franz, S. Scherzinger, et al., Peel — pile? cross-framework portability of quantum soft- ware, inProc. ICSA-C, 2022

    2022

  38. [38]

    Vallet, D

    N. Vallet, D. Michonneau, and S. Tournier, Toward practical transparent verifiable and long-term repro- ducible research using guix,Scientific Data, 2022

    2022

  39. [39]

    Court `es and R

    L. Court `es and R. Wurmus, Reproducible and User- Controlled Software Environments in HPC with Guix, inProc. Euro-Par Workshops, 2015

    2015

  40. [40]

    Hausch, S

    M. Hausch, S. Hauser, and B. Uekermann, Improving reproducibility of scientific software using nix/nixos: A case study on the precice ecosystem,ECEASST, 2025

    2025

  41. [41]

    Dolstra, M

    E. Dolstra, M. De Jonge, E. Visser, et al., Nix: A safe and policy-free system for software deployment. InLISA, 2004

    2004