REVIEW 2 major objections 4 minor
Only about a quarter of quantum computing papers share runnable code, and most of those packages fail in a clean environment.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 08:56 UTC pith:JM3R6BQA
load-bearing objection Solid empirical baseline: ~25% code availability and ~65% clean-environment failure in QC papers, with manual and automated numbers that line up and a concrete failure taxonomy. the 2 major comments →
Works on My QPU: Reproducibility in Quantum Computing Research
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Reproducibility is not yet consistently achieved in quantum computing research. Across a manual sample of 127 papers only 24.4 percent provide code artefacts, and 64.5 percent of those artefacts fail to run in a clean environment; a large-scale automated screen of nearly 5000 papers yields a matching code-availability rate of 26.8 percent and shows that roughly one-third of accessible repositories lack machine-readable environment specifications.
What carries the argument
A five-question reproducibility assessment framework (code availability, environment specification, documentation, hardware description, and executability) applied first by manual validation on a filtered sample and then by automated full-text and repository screening on a much larger corpus.
Load-bearing premise
The multi-stage filters used to build the manual sample (NISQ keyword, DOI as peer-review proxy, presence in both literature databases, experimental-language keywords) do not systematically bias code-sharing or executability rates away from the broader quantum computing literature.
What would settle it
Re-run the same five-question framework on a new random sample of recent experimental quantum computing papers drawn without the NISQ or dual-database filters; if code availability substantially exceeds roughly 25 percent and clean-environment success substantially exceeds roughly 35 percent, the central empirical claim fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a combined manual and automated empirical study of reproducibility practices in quantum computing research. From a multi-stage filtered sample of 127 NISQ-era experimental papers (drawn from a 4966-paper corpus spanning 2021–2026), the authors apply a five-question framework (code availability, environment specification, documentation, hardware specification, executability). Only 24.4% of the sample provide accessible code; of those packages, 64.5% fail clean-environment execution. An automated screen of the full corpus yields a consistent code-availability rate of 26.8% and shows that roughly one-third of repositories with code lack machine-readable environment files. Six recurring failure modes are catalogued, and the authors supply practical recommendations plus a reusable reproduction package (Makefile/Docker plus an exploratory Nix flake) that operationalises those recommendations.
Significance. If the measured rates hold, the work supplies the first large-scale, dual-method quantification of artefact availability and short-term executability in QC software research. The close agreement between the curated manual sample (24.4%) and the unfiltered automated corpus (26.8%) strengthens the claim that the headline figures are not artefacts of the NISQ/DOI/dual-database filters. The explicit failure taxonomy and the shipped analysis pipeline (GitHub) make the results independently re-verifiable and give the community concrete, actionable targets. The recommendations and template package further convert diagnosis into practice, which is of immediate value for authors, reviewers, and programme committees.
major comments (2)
- Section II-C.5 (RQ5) and the six failure modes: the executability claim rests on 31 packages, of which only a subset satisfied both RQ2 and RQ3 before the clean-environment attempt. The paper should state the exact number of packages that were actually executed (and the success/failure counts for each of the six modes) so that readers can judge the statistical weight of the 64.5% figure. A short table or appendix listing the 31 repositories and their RQ2–RQ5 outcomes would make the result fully auditable without altering the central claim.
- Section II-B.2 and Fig. 1: the multi-stage filter (NISQ keyword, DOI proxy, dual-database presence, experimental-language keywords) is acknowledged, yet the paper never reports a sensitivity check that relaxes any single filter and recomputes the code-availability rate. While the automated 4966-paper screen already mitigates the most serious bias concern, a one-paragraph sensitivity note (or a supplementary table) would close the residual sampling objection cleanly.
minor comments (4)
- Table II: the “N/A” column mixes “no code available” with “not applicable because classical simulation only.” A footnote clarifying the two meanings would improve readability.
- Section IV: the Nix flake is described as “exploratory.” A single sentence stating whether the flake was used to re-execute any of the 31 packages (or only the authors’ own pipeline) would set expectations correctly.
- References [36] and [17] are arXiv preprints dated 2026; ensure the final version cites the most recent stable identifiers or DOIs if they become available.
- Fig. 1 pipeline diagram: the arrow labels “249” and “127” are slightly hard to parse at a glance; adding the filter names next to the numbers would help.
Circularity Check
No significant circularity: headline rates are direct empirical counts from PDF/repository screening, not derived by construction from fitted inputs or load-bearing self-citations.
full rationale
The paper's central claims (code availability 24.4%/26.8%, ~64.5% of available packages failing clean-environment execution, ~1/3 lacking machine-readable environment specs) are obtained by keyword/full-text screening of arXiv+Semantic Scholar corpora, manual validation of a 127-paper NISQ subset, and automated checks on ~4966 papers (Section II, Fig. 1, Tab. II). These are observational tallies of presence/absence of artefacts and of whether packages ran under the authors' clean-environment protocol; they do not rest on any equation that defines a quantity in terms of itself, any parameter fitted to a subset then re-presented as a prediction, or any uniqueness theorem. Self-citations (e.g., prior Docker/Nix reproducibility templates by overlapping authors) appear only as background motivation and as the authors' own recommended mitigation; they are not used to force or justify the measured rates. The multi-stage filter is a sampling choice whose possible bias is cross-checked by the unfiltered large-scale run yielding a nearly identical availability rate; that check is independent evidence, not circular. No step reduces by construction to its inputs. Honest non-finding: score 0, empty steps.
Axiom & Free-Parameter Ledger
axioms (3)
- domain assumption Public code plus a machine-readable environment specification is a necessary precondition for computational reproducibility of QC software experiments.
- domain assumption A DOI is a reasonable proxy for peer review when filtering the paper corpus.
- ad hoc to paper Keyword presence of 'NISQ', 'experiment', 'numerical', etc., sufficiently isolates experimental QC papers for the manual sample.
read the original abstract
Quantum computing research increasingly depends on complex software stacks, yet the reproducibility of published results does not receive the priority and longevity mandated by recommendations of large international scientific bodies and best practices in software-centric systems research. In this paper, we present a combined manual and automated large-scale analysis of the reproducibility landscape in quantum computing research, quantify shortcomings, and derive actionable steps forward. We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability. To place these findings in a broader context, we conduct an automated large-scale screening of nearly 5000 quantum computing papers for the same reproducibility indicators. Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment. This assessment is corroborated by a large-scale automated analysis that yields a consistent code availability rate of 26.8%. Further, it shows that approximately one-third of the papers with accessible code lack machine-readable environment specifications. The results in this paper indicate that reproducibility is not yet consistently achieved in quantum computing research. In response, we outline a set of practical recommendations that address the observed failure modes and illustrate how reproducibility can be improved in practice.
Figures
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.