arxiv: 2604.25478 · v1 · submitted 2026-04-28 · 💻 cs.ET · cs.PL

Recognition: unknown

Practical Insights into Fair Comparison and Evaluation Frame for Neutral-Atom Compilers

Emil Khusainov , Yanbin Chen , Jonas Winklmann , Helmut Seidl , Christian B. Mendl

Authors on Pith no claims yet

Pith reviewed 2026-05-07 13:46 UTC · model grok-4.3

classification 💻 cs.ET cs.PL

keywords neutral-atom quantum computingquantum compilersfair evaluationRSQASMHybridMapperDasAtomEnola

0 comments

The pith

A unified RSQASM evaluation framework shows that many claimed performance gaps between neutral-atom compilers shrink or disappear under consistent conditions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neutral-atom quantum computing needs compilers that map qubits, route operations, schedule gates, and handle shuttling and parallel execution. Different compilers have been judged using mismatched metrics, varying movement-duration models, inconsistent fidelity sources, and sometimes undocumented choices, so reported differences could not be trusted. The paper supplies RSQASM, a common post-compilation format that records all these elements explicitly, plus adapter scripts that convert the outputs of existing compilers into this single representation. When the three compilers previously said to differ by large margins are re-evaluated inside this framework, the gaps become substantially smaller and in some cases vanish. This matters because only consistent measurements can tell developers which compiler improvements are real and worth pursuing for scalable hardware.

Core claim

We present a unified and reproducible evaluation framework for neutral-atom compilers. Our framework introduces RSQASM, a QASM-inspired post-compilation representation that captures mapped, routed, and scheduled circuits, including explicit parallel gate execution and shuttling operations. As part of the framework, we provide adapter scripts that translate existing compiler outputs and intermediate artifacts into RSQASM. As a case study, we compare three well-known neutral-atom compilation toolchains: HybridMapper, DasAtom, and Enola. Using our framework and representation, we perform a new evaluation and show that several previously claimed performance gaps become substantially smaller and,

What carries the argument

RSQASM, a QASM-inspired post-compilation representation that captures mapped, routed, and scheduled circuits including explicit parallel gate execution and shuttling operations, together with adapter scripts that convert each compiler's outputs into this common format.

If this is right

Performance numbers from different compilers can be compared directly without confounding differences in transpilation levels or movement models.
Some previously reported advantages of one toolchain over others may not survive once evaluation inconsistencies are removed.
Future compiler papers can adopt the same representation and adapters to produce reproducible, side-by-side results.
Developers can focus effort on genuine improvements rather than artifacts of mismatched measurement choices.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Other quantum hardware platforms with similar compilation steps could adopt an analogous standardized representation to enable fair cross-compiler studies.
Open-sourcing the adapter scripts would let independent groups verify that no hidden biases remain in the translation step.
Extending the framework to include actual hardware noise models would turn the comparison from abstract metrics into predicted execution fidelity.

Load-bearing premise

The adapter scripts faithfully translate each compiler's internal decisions into RSQASM without introducing new biases or omitting undocumented representation choices that affected the original reported metrics.

What would settle it

Reproduce the original papers' metrics with their reported inconsistent settings and then re-run the same compilers through the RSQASM adapters; if the large gaps reappear in the first case but shrink in the second, the claim is supported.

Figures

Figures reproduced from arXiv: 2604.25478 by Christian B. Mendl, Emil Khusainov, Helmut Seidl, Jonas Winklmann, Yanbin Chen.

**Figure 1.** Figure 1: Overview of the proposed benchmarking framework. view at source ↗

**Figure 2.** Figure 2: Detailed overview of the proposed benchmarking view at source ↗

read the original abstract

Neutral-atom quantum computing is among the most promising platforms for scalable quantum computation, and compilation toolchains are crucial for leveraging capabilities such as qubit shuttling and parallel gate execution. An important challenge, however, is that existing neutral-atom compilers are often evaluated using metrics computed over different parts of the toolchain and under non-equivalent assumptions. Consequently, fair quantification and comparison of compiler performance remain difficult. Reported metrics may depend on inconsistent transpilation optimization levels, different movement-duration models, different sets of considered fidelity sources, and even minor implementation bugs or undocumented representation choices. To address this problem, we present a unified and reproducible evaluation framework for neutral-atom compilers. Our framework introduces RSQASM (Routed and Scheduled QASM), a QASM-inspired post-compilation representation that captures mapped, routed, and scheduled circuits, including explicit parallel gate execution and shuttling operations. As part of the framework, we provide adapter scripts that translate existing compiler outputs and intermediate artifacts into RSQASM. As a case study, we compare three well-known neutral-atom compilation toolchains: HybridMapper, DasAtom, and Enola, motivated by the large performance differences reported in prior work. Using our framework and representation, we perform a new evaluation and show that several previously claimed performance gaps become substantially smaller and, in some cases, are not reproduced once evaluation inconsistencies are removed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives neutral-atom compiler researchers a shared RSQASM representation plus adapters so they can re-run comparisons under consistent rules, and it shows that some earlier performance gaps shrink once those rules are applied.

read the letter

The core contribution is RSQASM, a post-compilation format that records mapping, routing, scheduling, and parallel gate execution in one place. The authors supply adapter scripts that turn outputs from HybridMapper, DasAtom, and Enola into this format, then re-evaluate the three tools on the same footing. The result is that several previously reported gaps become smaller or disappear, which the abstract attributes to differences in transpilation levels, movement models, and fidelity accounting across the original papers.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces a unified evaluation framework for neutral-atom compilers using RSQASM, a QASM-inspired representation that captures mapped, routed, and scheduled circuits with explicit parallel gates and shuttling. Adapters are provided to translate outputs from HybridMapper, DasAtom, and Enola into this format. The case study re-evaluation demonstrates that previously reported performance gaps between these compilers become substantially smaller or are not reproduced when inconsistencies in transpilation, movement models, and fidelity sources are removed.

Significance. This work addresses a significant practical issue in the neutral-atom quantum computing community by promoting standardized and reproducible evaluations. If the adapters prove faithful, the framework could become a standard tool for fair benchmarking, leading to more accurate assessments of compiler performance and accelerating progress in the field. The emphasis on removing evaluation inconsistencies is a strength, and making the framework available would be a positive contribution.

major comments (2)

The key claim that performance gaps shrink once inconsistencies are removed rests on the fidelity of the adapter scripts. The abstract indicates that adapters translate 'existing compiler outputs and intermediate artifacts into RSQASM,' but the manuscript must demonstrate that these translations reproduce the original reported metrics exactly for the same input circuits. Without such verification, it is impossible to rule out that the reduced gaps result from changes introduced by the adapters rather than the standardization itself. This is load-bearing for the central claim.
The paper should include a direct comparison table of original vs. re-evaluated metrics (e.g., for shuttling duration, fidelity) for each compiler on the benchmark set to quantify the gap reduction and allow readers to assess the impact.

minor comments (2)

Provide pseudocode or a detailed description of how RSQASM encodes parallel gate execution and shuttling operations to improve clarity.
Ensure all benchmark circuits used in the case study are listed or made available for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which identify key areas to strengthen the evidence supporting our claims about the evaluation framework. We address each major comment below and will incorporate the suggested revisions to improve the manuscript.

read point-by-point responses

Referee: The key claim that performance gaps shrink once inconsistencies are removed rests on the fidelity of the adapter scripts. The abstract indicates that adapters translate 'existing compiler outputs and intermediate artifacts into RSQASM,' but the manuscript must demonstrate that these translations reproduce the original reported metrics exactly for the same input circuits. Without such verification, it is impossible to rule out that the reduced gaps result from changes introduced by the adapters rather than the standardization itself. This is load-bearing for the central claim.

Authors: We agree that explicit verification of the adapter scripts is necessary to substantiate the central claim. The current manuscript describes the adapters and their role in translating compiler outputs but does not provide a direct demonstration that they reproduce the original metrics under the source papers' assumptions. In the revised manuscript, we will add a dedicated verification subsection (e.g., within Section 4) that applies the adapters to original compiler artifacts for representative benchmark circuits, recomputes the metrics in RSQASM, and confirms exact matches to the originally reported values when using identical transpilation levels, movement models, and fidelity sources. This will establish that the adapters are faithful and that any reductions in performance gaps result from the standardized evaluation rather than adapter-induced changes. revision: yes
Referee: The paper should include a direct comparison table of original vs. re-evaluated metrics (e.g., for shuttling duration, fidelity) for each compiler on the benchmark set to quantify the gap reduction and allow readers to assess the impact.

Authors: We thank the referee for this suggestion, which will improve the clarity and interpretability of our results. We will add a new comparison table (e.g., Table 3) in the case study section. For each compiler and each circuit in the benchmark set, the table will report the original published metrics (shuttling duration, fidelity, etc.) alongside the re-evaluated metrics obtained via the RSQASM framework under consistent assumptions. It will also include the absolute and relative gap reductions between compilers. This will allow readers to directly quantify the effect of removing evaluation inconsistencies. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework and re-evaluation rest on independent adapter translations and external prior metrics

full rationale

The paper introduces RSQASM as a post-compilation representation and provides adapter scripts to translate outputs from existing compilers (HybridMapper, DasAtom, Enola) into this format, then performs a case-study re-evaluation showing that previously reported performance gaps shrink under standardized metrics. No equations, fitted parameters, or derivations are claimed; the central result is an empirical comparison that relies on the adapters faithfully reproducing original compiler decisions. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes, and the adapters are presented as direct translators rather than tuned models. The evaluation is therefore self-contained against the external benchmarks from prior work, with no reduction of outputs to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework assumes that movement-duration models, fidelity sources, and transpilation levels can be standardized without loss of essential compiler behavior; no new physical entities or free parameters are introduced beyond the representation itself.

axioms (1)

domain assumption Existing compiler outputs can be losslessly translated into a common routed-and-scheduled representation without altering the underlying scheduling decisions.
Invoked when adapter scripts are used to convert HybridMapper, DasAtom, and Enola artifacts into RSQASM.

pith-pipeline@v0.9.0 · 5554 in / 1355 out tokens · 53069 ms · 2026-05-07T13:46:59.987194+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

17 extracted references · 9 canonical work pages

[1]

Neutral atom quantum computing hardware: performance and end-user perspective,

K. Wintersperger, F. Dommert, T. Ehmer, A. Hoursanov, J. Klepsch, W. Mauerer, G. Reuber, T. Strohm, M. Yin, and S. Luber, “Neutral atom quantum computing hardware: performance and end-user perspective,” EPJ Quantum Technology, vol. 10, no. 1, Aug. 2023. [Online]. Available: http://dx.doi.org/10.1140/epjqt/s40507-023-00190-1

work page doi:10.1140/epjqt/s40507-023-00190-1 2023
[2]

A quantum processor based on coherent transport of entangled atom arrays,

D. Bluvstein, H. Levine, G. Semeghini, T. T. Wang, S. Ebadi, M. Kali- nowski, A. Keesling, N. Maskara, H. Pichler, M. Greiner, V . Vuleti ´c, and M. D. Lukin, “A quantum processor based on coherent transport of entangled atom arrays,”Nature, vol. 604, no. 7906, pp. 451–456, Apr. 2022

2022
[3]

Multi-qubit entanglement and algorithms on a neutral-atom quantum computer,

T. M. Graham, Y . Song, J. Scott, C. Poole, L. Phuttitarn, K. Jooya, P. Eichler, X. Jiang, A. Marra, B. Grinkemeyer, M. Kwon, M. Ebert, J. Cherek, M. T. Lichtman, M. Gillette, J. Gilbert, D. Bowman, T. Bal- lance, C. Campbell, E. D. Dahl, O. Crawford, N. S. Blunt, B. Rogers, T. Noel, and M. Saffman, “Multi-qubit entanglement and algorithms on a neutral-at...

2022
[4]

Assembly and coherent control of a register of nuclear spin qubits,

K. Barnes, P. Battaglino, B. J. Bloom, K. Cassella, R. Coxe, N. Crisosto, J. P. King, S. S. Kondov, K. Kotru, S. C. Larsen, J. Lauigan, B. J. Lester, M. McDonald, E. Megidish, S. Narayanaswami, C. Nishiguchi, R. Notermans, L. S. Peng, A. Ryou, T.-Y . Wu, and M. Yarwood, “Assembly and coherent control of a register of nuclear spin qubits,” Nature Communica...

2022
[5]

Quantum computing with neutral atoms,

L. Henriet, L. Beguin, A. Signoles, T. Lahaye, A. Browaeys, G.-O. Reymond, and C. Jurczak, “Quantum computing with neutral atoms,”Quantum, vol. 4, p. 327, Sep. 2020. [Online]. Available: https://doi.org/10.22331/q-2020-09-21-327

work page doi:10.22331/q-2020-09-21-327 2020
[6]

Hybrid circuit mapping: Leveraging the full spectrum of computational capabilities of neutral atom quantum computers,

L. Schmid, S. Park, S. Kang, and R. Wille, “Hybrid circuit mapping: Leveraging the full spectrum of computational capabilities of neutral atom quantum computers,” 2023. [Online]. Available: https://arxiv.org/abs/2311.14164

work page arXiv 2023
[7]

Dasatom: A divide-and-shuttle atom approach to quantum circuit transformation,

Y . Huang, D. Gao, S. Ying, and S. Li, “Dasatom: A divide-and-shuttle atom approach to quantum circuit transformation,” 2025. [Online]. Available: https://arxiv.org/abs/2409.03185

work page arXiv 2025
[8]

D. B. Tan, W.-H. Lin, and J. Cong,Compilation for Dynamically Field-Programmable Qubit Arrays with Efficient and Provably Near-Optimal Scheduling. New York, NY , USA: Association for Computing Machinery, 2025, p. 921–929. [Online]. Available: https: //doi.org/10.1145/3658617.3697778

work page doi:10.1145/3658617.3697778 2025
[9]

Atomique: A quantum compiler for reconfigurable neutral atom arrays,

H. Wang, P. Liu, D. B. Tan, Y . Liu, J. Gu, D. Z. Pan, J. Cong, U. A. Acar, and S. Han, “Atomique: A quantum compiler for reconfigurable neutral atom arrays,” 2024. [Online]. Available: https://arxiv.org/abs/2311.15123

work page arXiv 2024
[10]

A review and collection of metrics and benchmarks for quantum computers: definitions, methodologies and software,

D. Lall, A. Agarwal, W. Zhang, L. Lindoy, T. Lindstr ¨om, S. Webster, S. Hall, N. Chancellor, P. Wallden, R. Garcia-Patron, E. Kashefi, V . Kendon, J. Pritchard, A. Rossi, A. Datta, T. Kapourniotis, K. Georgopoulos, and I. Rungger, “A review and collection of metrics and benchmarks for quantum computers: definitions, methodologies and software,” 2025. [On...

work page arXiv 2025
[11]

Tooling and benchmarking of a hardware-agnostic compilation toolchain for neutral-atom quantum computers,

E. Khusainov, “Tooling and benchmarking of a hardware-agnostic compilation toolchain for neutral-atom quantum computers,” 2025. [Online]. Available: https://mediatum.ub.tum.de/node?id=1788052& change language=en

2025
[12]

Computational capabilities and compiler development for neutral atom quantum processors—connecting tool developers and hardware experts,

L. Schmid, D. F. Locher, M. Rispler, S. Blatt, J. Zeiher, M. M ¨uller, and R. Wille, “Computational capabilities and compiler development for neutral atom quantum processors—connecting tool developers and hardware experts,”Quantum Science and Technology, vol. 9, no. 3, p. 033001, Apr. 2024. [Online]. Available: http://dx.doi.org/10.1088/ 2058-9565/ad33ac

2024
[13]

Compiling quantum circuits for dynamically field-programmable neutral atoms array processors,

D. B. Tan, D. Bluvstein, M. D. Lukin, and J. Cong, “Compiling quantum circuits for dynamically field-programmable neutral atoms array processors,”Quantum, vol. 8, p. 1281, Mar. 2024. [Online]. Available: http://dx.doi.org/10.22331/q-2024-03-14-1281 [14]IBM Qiskit Compiler Transpile Documentation, IBM Qiskit, 2026, accessed: 2026-03-03 for version 2.4.0. [...

work page doi:10.22331/q-2024-03-14-1281 2024
[14]

qiskit/compiler/transpiler.py line 224 (state by last commit 7d29dc1b33ab0229bc77ab773b8d2f7fd2589552) for release 1.1.0,

Qiskit Development Team, “qiskit/compiler/transpiler.py line 224 (state by last commit 7d29dc1b33ab0229bc77ab773b8d2f7fd2589552) for release 1.1.0,” 2026, accessed: 2026-03-

2026
[15]

Available: https://github.com/Qiskit/qiskit/commit/ 7d29dc1b33ab0229bc77ab773b8d2f7fd2589552

[Online]. Available: https://github.com/Qiskit/qiskit/commit/ 7d29dc1b33ab0229bc77ab773b8d2f7fd2589552
[16]

Enola/run qasm.py line 25 (state at commit 2944dbf4e163e8d2eeeec607add0d9139edce689),

UCLA-V AST Development Team, “Enola/run qasm.py line 25 (state at commit 2944dbf4e163e8d2eeeec607add0d9139edce689),” 2026, accessed: 2026-03-05. [Online]. Avail- able: https://github.com/UCLA-V AST/Enola/commit/ 2944dbf4e163e8d2eeeec607add0d9139edce689

2026
[17]

Metriq: A collaborative platform for benchmarking quantum computers.arXiv preprint arXiv:2603.08680, 2026

A. Cosentino, C. Li, V . Russo, B. A. Chase, T. Lubinski, S. Niu, N. Patel, N. Shammah, and W. J. Zeng, “Metriq: A collaborative platform for benchmarking quantum computers,” 2026. [Online]. Available: https://arxiv.org/abs/2603.08680

work page arXiv 2026