REVIEW 3 major objections 2 minor 1 cited by
Benchmarking quantum computers with any quantum algorithm
T0 review · 3 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that running many small subcircuits snipped from any quantum algorithm yields a scalable benchmark, summarized by a capability coefficient that tracks progress toward executing the full algorithm.
desk verdict Abstract-only review of a promising benchmarking method; the core extrapolation claim needs evidence that isn't in the abstract. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Subcircuit volumetric benchmarking (SVB): the procedure of generating a distribution of subcircuits from a target algorithm, executing a sample of them on a quantum computer, and aggregating the outcomes into a capability coefficient. The target circuit serves both as the workload generator and as the yardstick, so the benchmark measures progress toward that specific algorithm.
What would settle it
On a device small enough to execute a full target circuit, compute the SVB capability coefficient from its subcircuits, then run the full circuit and measure its true success probability; if the coefficient does not rank devices consistently with full-circuit fidelity, the representativeness assumption is refuted.
Extended reading notes
Core claim
The central claim is that any quantum algorithm can serve as a benchmark workload. Given a target circuit—potentially a utility-scale application—SVB defines a distribution of subcircuits by 'snipping out' contiguous pieces of that circuit. The hardware is benchmarked by running an efficiently chosen sample of these subcircuits, spanning a range of sizes and shapes, and the results are compressed into a single capability coefficient. SVB's scalability follows because the number and size of subcircuits needed does not grow with the target circuit itself. A demonstration on IBM Q hardware, using a Hamiltonian block-encoding subroutine from quantum chemistry, illustrates the method.
Load-bearing premise
The load-bearing assumption is that how a quantum computer performs on small subcircuits snipped from a target circuit is representative of how it would perform on the full target circuit, since the benchmark never runs the full circuit.
Editorial extensions
If this is right
- Benchmarks can be constructed from any quantum algorithm, not just from small hand-picked test problems.
- Progress toward utility-scale applications can be tracked without running utility-scale circuits.
- Different quantum computers can be compared by their capability coefficient on the same target circuit.
- The method is scalable and efficient, with benchmarking cost controlled by the subcircuit distribution rather than the full circuit size.
Reading between the lines
- A natural extension, not stated in the abstract, is to use the capability coefficient as a predictor of full-circuit success probability; if validated, it could become a standard figure of merit for hardware comparisons.
- The abstract does not discuss how to choose the subcircuit sampling distribution; optimizing it to minimize the number of runs for a given confidence would be a direct follow-up.
- Because SVB reports outcomes on many subcircuits, it could be combined with error-mitigation techniques to identify which gate types or circuit regions are the current bottleneck for a particular target.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Subcircuit Volumetric Benchmarking (SVB), a method for constructing scalable and efficient benchmarks from any quantum algorithm or application. The central idea is to run subcircuits of varied shape that are 'snipped out' from a target circuit, such as a utility-scale algorithm, and to use the results to estimate a capability coefficient that summarizes progress toward implementing the target circuit. The authors report a demonstration on IBM Q systems using a Hamiltonian block-encoding subroutine from quantum chemistry. Because only the abstract was available for review, the evaluation is necessarily limited to the claims as stated.
Significance. If SVB delivers what the abstract promises, it would be a valuable contribution: it would allow any quantum algorithm to serve as a benchmark source, sidestepping the problem that current hardware cannot run utility-scale circuits directly. The claimed scalability and efficiency of the method, together with a concise capability coefficient, are potentially useful for tracking hardware progress. The explicit experimental demonstration on IBM Q is a positive sign. However, the abstract alone provides no derivations, no definitions, and no supporting data, and the central extrapolation assumption is unstated and unjustified in the visible text.
major comments (3)
- [Abstract (central claim)] The abstract claims that SVB 'enables estimating a capability coefficient that concisely summarizes progress towards implementing the target circuit.' This requires the assumption that performance on snipped subcircuits is predictive of performance on the full target circuit. No argument—statistical, analytical, or empirical—is given for this representativeness. Realistic error mechanisms such as crosstalk, spectator-qubit errors, correlated dephasing, or coherent errors that only accumulate at full depth/width may not appear in small subcircuits. This is a load-bearing point: without a demonstration of subcircuit-to-full-circuit extrapolation, the coefficient cannot be interpreted as summarizing progress toward the target circuit. The paper must either provide a validation on circuits where the full-circuit success probability is known, or carefully restrict the claim to what the subcir
- [Abstract (definitions)] The capability coefficient is not defined, and no equation or procedure is given for estimating it from subcircuit runs. Without a precise definition, the claim that it 'concisely summarizes progress' is not testable. The method's scalability and efficiency are also stated without any algorithmic description; no complexity argument, no sampling strategy, and no error model are presented. These omissions prevent assessment of whether the method is genuinely scalable or merely applied to a single example. At minimum, the full paper must supply the formal definitions and complexity analysis.
- [Abstract (experimental demonstration)] The demonstration on IBM Q is mentioned but no details are given: no circuits, no error bars, no comparison against a known ground truth, no control experiments, and no analysis of whether the measured capability coefficient tracks actual full-circuit performance. The experiment appears to show that SVB is implementable, but it does not validate the central predictive claim. The full paper should include such a validation; without it, the experimental section cannot support the abstract's broad conclusions.
minor comments (2)
- [Abstract] The term 'subcircuit volumetric benchmarking' is introduced but its relationship to prior volumetric benchmarking (e.g., Quantum Volume) is not clarified; a brief comparison would help situate the method.
- [Abstract] The phrase 'of varied shape' is vague. It is unclear whether shape refers to width, depth, aspect ratio, or a combination. Defining this explicitly would improve precision.
Circularity Check
No circularity identified in the abstract; the subcircuit-to-full-circuit relationship is an extrapolation assumption, not a definitional or fitted equivalence.
full rationale
The abstract describes SVB as a method that runs subcircuits 'snipped out' from a target circuit and estimates a capability coefficient summarizing progress toward that target. This is not circular: the coefficient is defined from measurements on the subcircuits, not fitted to the target circuit's output, and the subcircuits are not constructed from the coefficient. The concern that subcircuit performance may not predict full-circuit performance is a representativeness or validity assumption, not a circularity. No equation is provided in the abstract that would let one exhibit a reduction of the claimed prediction to its inputs, and no load-bearing self-citation is visible. The paper's central claim may be unvalidated without additional evidence, but that is a correctness or support concern, not circular reasoning. Therefore, the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Subcircuit performance is representative of full-circuit performance
- domain assumption The sampled subcircuits cover the target circuit's computational behavior
Cite this review
Pith. "Pith review of Benchmarking quantum computers with any quantum algorithm." pith.science (2026). https://pith.science/paper/2N4NVZTB
@misc{pith2026250805754,
author = {Pith},
title = {Pith review of: Benchmarking quantum computers with any quantum algorithm},
year = {2026},
howpublished = {\url{https://pith.science/paper/2N4NVZTB}},
note = {Machine review of arXiv:2508.05754}
}
read the original abstract
Application-based benchmarks are increasingly used to quantify and compare quantum computers' performance. However, because contemporary quantum computers cannot run utility-scale computations, these benchmarks currently test this hardware's performance on ``small'' problem instances that are not necessarily representative of utility-scale problems. Furthermore, these benchmarks often employ methods that are unscalable, limiting their ability to track progress towards utility-scale applications. In this work, we present a method for creating scalable and efficient benchmarks from any quantum algorithm or application. Our subcircuit volumetric benchmarking (SVB) method runs subcircuits of varied shape that are ``snipped out'' from some target circuit, which could implement a utility-scale algorithm. SVB is scalable and it enables estimating a capability coefficient that concisely summarizes progress towards implementing the target circuit. We demonstrate SVB with experiments on IBM Q systems using a Hamiltonian block-encoding subroutine from quantum chemistry algorithms.
Forward citations
Cited by 1 Pith paper
-
Clifford Volume and Free Fermion Volume: Complementary Scalable Benchmarks for Quantum Computers
Two new classically verifiable benchmark scores, Clifford Volume and Free Fermion Volume, are defined, simulated under noise, and Clifford Volume is measured on the Quantinuum H2-1 device as 34 qubits.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.