Pith. sign in

REVIEW 5 major objections 7 minor 25 references

First Experience with Real-Time Control Using Simulated VQC-Based Quantum Policies

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A purely variational quantum circuit, trained offline as a policy, can balance a physical cart-pole when executed on a fast local simulator, but cloud quantum execution is too slow for real-time control.

desk verdict Deploying a VQC policy on real hardware is a useful first step, but the bin-averaged success metric in Figure 6 undercuts the central claim until per-episode data are reported. read the letter →

arxiv 2508.01690 v2 pith:IRSPMM5F submitted 2025-08-03 quant-ph

Yize Sun , Mohamad Hagog , Marc Weber , Daniel Hein , Steffen Udluft , Volker Tresp , Yunpu Ma This is my paper · ORCID
classification quant-ph
keywords variationalquantumcircuitsreinforcementlearningofflinemodel-basedpolicysearchcart-polecontrolreal-timesim-to-realtransferlatencyanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a quantum circuit can serve as the decision-making policy in a real-time control loop, not just in simulation. The authors train a variational quantum circuit (VQC) purely offline, using model-based policy search with an ensemble of learned dynamics models, and then deploy it on a physical cart-pole. They report that the trained quantum policy balances the cart for more than 200 steps in most starting-position bins, and that local simulated inference takes about 4 ms, well within the 15 ms real-time budget. Cloud-based quantum execution, by contrast, takes about 3.7 seconds per action, ruling it out for closed-loop control. The result matters because it shows a quantum model with no classical neural-network layers can produce usable continuous control from offline data, while identifying latency on real quantum hardware as the current bottleneck.

What carries the argument

The load-bearing object is the VQC policy: eight qubits, each input component $x_i$ encoded as a trainable rotation $R_X(w_i x_i)$, followed by two strongly entangling layers of $R_Z$–$R_Y$–$R_Z$ single-qubit rotations with CNOT gates in a ring, and a computational-basis measurement scaled by a trainable output weight $w_{\mathrm{out}}$. This circuit replaces the classical policy network inside MOOSE, so policy gradients flow through fixed ensemble transition models during offline training. The trainable input weights let the circuit learn how to embed each feature, and the output weight maps the quantum expectation value to a continuous action without any classical postprocessing layer; this design is what makes the pure quantum policy trainable enough to transfer to hardware.

What would settle it

Re-running the trained QMOOSE policy on the physical cart-pole with many episodes from every 10-unit start-position bin, while measuring per-action inference latency on the same local hardware setup, would settle the claim: the central result fails if most bins do not average 200 balancing steps or if the local inference time consistently exceeds 15 ms.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a pure variational quantum circuit, with all trainable parameters inside the circuit, can learn a cart-pole balancing policy from a fixed offline dataset and transfer that policy to real hardware. The policy, called QMOOSE, replaces the neural network in the MOOSE offline reinforcement learning algorithm; it encodes the eight-dimensional state through trainable rotations, passes it through two strongly entangling layers, and scales the measured expectation value by a trainable output weight. In hardware evaluation, most track-position bins average at least 200 balancing steps, meeting the authors' success criterion, while local simulated inference at 4±1 ms satisfies the 15 ms real-time constraint. The same policy without trainable input-output weights fails to learn, and cloud-based QPU execution adds roughly 3.7 seconds per action, so the paper concludes that only local simulation is currently viable for latency-sensitive control.

Load-bearing premise

The whole result rests on the offline dataset and the learned ensemble of dynamics models being accurate enough that a policy trained inside them works on the real cart-pole; the authors' own comparison shows the surrogate overestimates hardware performance, so if the data or models are biased, the edge failures in hardware follow.

Editorial extensions

If this is right

  • If the central claim holds, purely quantum policies can be trained offline and deployed in real-time control, so quantum models are not confined to simulation benchmarks.
  • Trainable input and output weights become a necessary ingredient for this circuit class: without them the same architecture cannot learn the balancing task.
  • Real-time closed-loop control with cloud-based quantum processors is ruled out at current latencies (about 3.7 s per action), so near-term deployments must use local simulation or reduce end-to-end quantum latency by orders of magnitude.
  • The sim-to-real gap shown in the hardware evaluation means model-based offline training should be tested across the full state space, especially edge regions, before trusting surrogate evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 200-step threshold at a 20 ms loop spans only about 4 seconds of balancing, the result should be read as a feasibility demonstration; longer-horizon industrial benchmarks would likely expose the edge-region failures more sharply.
  • The success of trainable input scaling suggests a general recipe for VQC policies in continuous control: let the encoding and the output scale be learned parameters, independent of the circuit depth.
  • The latency breakdown implies a design rule for quantum control: a plant whose control period exceeds the end-to-end quantum inference time could, in principle, be driven by a real QPU; the cart-pole's 15–20 ms period is far too fast.
  • Since the circuit is simulated rather than run on quantum hardware, the demonstration isolates algorithmic feasibility; real-device noise and measurement overhead would likely degrade both training transfer and latency, so the positive result is not yet evidence of a quantum advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper presents an offline model-based reinforcement learning pipeline in which the policy is a variational quantum circuit (VQC) with trainable input-scaling and output weights, called QMOOSE. The policy is trained on a fixed dataset collected from a physical cart-pole system, using an ensemble of learned transition models to generate synthetic rollouts. The trained quantum policy is then deployed on a Raspberry Pi/Arduino cart-pole setup, and the authors compare local simulated inference with remote IBM Cloud QPU execution. The central claims are that QMOOSE with trainable weights can balance the physical cart-pole for at least 200 steps in most start-position bins, that local simulated inference (4 +/- 1 ms) satisfies a 15 ms real-time budget, and that cloud QPU execution (about 3.7 s) does not.

Significance. If the central claims hold, this is a valuable empirical contribution: it is one of the few demonstrations of a pure VQC-based policy, trained via model-based offline RL, deployed on physical control hardware, and it provides a concrete latency characterization of local versus cloud quantum execution. The paper is also commendably candid about its limitations, including the sim-to-real gap and the restriction to simulated quantum circuits. However, the evidence as reported is incomplete. The hardware success criterion is defined per episode, but the only quantitative result is presented as bin averages without per-run counts, error bars, or success fractions; no classical MOOSE policy is run on the same hardware protocol; and key preprocessing and reward details are underspecified. These gaps currently prevent the reader from fully assessing the headline claim, but they are addressable within the scope of a revision.

major comments (5)
  1. [Sec. V-b, Fig. 6] The paper defines the baseline success criterion as the ability to balance the real cart-pole for at least 200 consecutive steps, but the only quantitative hardware evidence is Figure 6, which plots the average number of balancing steps over 10 runs per start-position bin. A bin average above 200 does not imply that individual episodes satisfy the criterion; one or two long runs can lift the mean while most runs fail early. Please report per-run counts, or at least medians, quartiles, and the fraction of runs exceeding 200 steps for each bin, along with confidence intervals. Without these, the central claim that QMOOSE 'can balance the hardware cart-pole' is not supported by the reported metric.
  2. [Sec. V-b, Table I] The evaluation section announces a comparison against classical baselines, but no classical MOOSE policy is run on the physical hardware under the same binned protocol. Table I reports only inference times, and Figure 5 is entirely in the learned surrogate. A classical MOOSE hardware baseline is needed to interpret the absolute 200-step threshold and to assess the relative value of the quantum policy; otherwise the reader cannot tell whether the QMOOSE hardware behavior reflects a meaningful control capability or merely the permissiveness of the test protocol.
  3. [Sec. IV-A] The preprocessing step that removes 'invalid or extreme observations, such as out-of-bounds positions or discontinuous angles' is never specified. The exact thresholds for out-of-bounds and discontinuity, and the fraction of the dataset removed, are load-bearing because the offline dataset determines both the ensemble transition model and the initial-state distribution used in Eq. (1). Without these details, the reported edge-case failures in Figure 6 cannot be attributed to data coverage, and the training pipeline cannot be reproduced.
  4. [Sec. V-a, Fig. 5] The surrogate evaluation compares classical MOOSE and QMOOSE using the same learned transition models that generated the training rollouts. This is a circular evaluation: the policy is optimized inside those models and then scored inside the same models, so the surrogate results are expected to be optimistic. The authors should state this limitation explicitly and, ideally, evaluate on held-out transition models or report disagreement across ensemble members. This does not invalidate the hardware result, but it weakens the motivation that 'positive results in the surrogate' justify deployment.
  5. [Sec. III-A and Sec. V-b, Table I] Section III-A states that the physical system operates with an average action loop of approximately 20 ms, while Section V-b defines real-time control as requiring inference below 15 ms and Table I reports QMOOSE inference at 4 +/- 1 ms. If the 20 ms figure is the actual closed-loop control period, then the deployed system exceeds the stated 15 ms budget; if the 15 ms budget applies only to policy inference rather than the full action loop, this distinction must be stated explicitly. The manuscript should clarify which number defines the real-time requirement and report the full action-loop latency on hardware, because the claim that 'local simulated execution meets real-time requirements' is load-bearing.
minor comments (7)
  1. [Sec. II-B] The statement that 'all trainable parameters are contained within the quantum circuit' is misleading: the weights w_i scale classical inputs before encoding, and w_out rescales the classical measurement outcome. The paper should say 'no classical neural network layers' rather than implying that all trainable parameters are quantum gates.
  2. [Sec. III-B-c] The reward function is described qualitatively but its coefficients are never given; a concrete equation with all coefficients is needed for reproducibility.
  3. [Table I] The final training losses are reported without noting that a lower numerical value for QMOOSE w/o TW (-18) is worse under the negative-reward convention; consider reporting returns or success rates to avoid confusion.
  4. [Table II] The QPU queue time is omitted due to variability; the paper should at least report a typical range, since queue time can dominate end-to-end cloud latency and the 3.7 s figure is therefore not a stable characterization.
  5. [Fig. 5] The shaded standard deviation bands are described as 'across repeated episodes', but the number of repeated episodes is only 5 and no success criterion is applied; please state whether all 5 runs completed the full 200 steps.
  6. [Sec. IV-B] The hyperparameter search is described, but the search ranges and the criterion for selecting the final configuration are not reported; this limits reproducibility of the Optuna results.
  7. [General] The paper would benefit from a data and code availability statement; the manuscript currently gives no indication that the offline dataset, preprocessing scripts, or trained circuit parameters will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central hardware-balancing claim is externally grounded in real cart-pole measurements, and self-citations are not load-bearing.

full rationale

The paper's central claim—that model-based offline policy search can generate quantum policies that balance the physical cart-pole—is supported by real hardware runs reported in Sec. V-b and Figure 6, which are external measurements not produced by the fitted transition model. The only quasi-circular component is the surrogate pre-evaluation in Sec. V-a and Figure 5, which evaluates policies in the same learned transition model used to generate training rollouts; that is a self-consistency check rather than an independent prediction. The authors explicitly acknowledge this limitation: 'this surrogate performance does not fully translate to robust real-world control,' and they do not base the central conclusion on the surrogate result. Self-citations [12], [15], and [21] introduce the base MOOSE algorithm and related prior work, but they are not used to derive or force the hardware outcome; there is no uniqueness theorem, no ansatz smuggled via citation, and no fitted parameter renamed as a prediction. The latency analysis (Tables I and II) is likewise an external measurement. Therefore, no circular step in the sense defined by the review criteria is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a trained policy, not a derivation, so the main ledger entries are fitted parameters and modeling assumptions rather than invented entities. The unstated reward coefficients and preprocessing thresholds are the most significant undocumented degrees of freedom.

free parameters (5)
  • VQC variational angles (theta, phi, delta) = not reported
    All rotation angles in the two strongly-entangling layers are trainable and fit by gradient descent on model-based rollouts (Sec II-A).
  • Input scaling weights w_i = not reported
    Feature-dependent scaling factors in the RX(w_i x_i) encoding are trained along with the circuit (Sec II-B).
  • Output weight w_out = not reported
    A trainable scalar multiplies the measured expectation value to produce the continuous action (Sec II-B).
  • Hyperparameters selected by Optuna = learning rate 0.01, 2 variational layers, data reuploading on, trainable input and output weights on
    200 Optuna trials selected these settings on the surrogate; this is fitting to the validation objective (Sec IV-B).
  • Reward function coefficients = not specified
    The reward penalizes position, angle, and angular velocity, but the exact coefficients and the angular velocity condition at +/-15 degrees are not stated, leaving hand-tuned degrees of freedom (Sec III-B-c).
assumptions (6)
  • domain assumption The offline dataset collected from the physical cart-pole is representative of the states and actions needed for control.
    Preprocessing removes invalid or extreme observations without stated thresholds, and training initial states are sampled from this dataset (Sec IV-A).
  • domain assumption The ensemble of K transition models accurately predicts real cart-pole dynamics for policy optimization.
    All policy training uses model-based rollouts from these fixed models (Sec III-B-a, Eq. 1), and the paper notes surrogate performance overestimates real performance (Sec V-b).
  • domain assumption Classical simulation of the VQC with TensorCircuit faithfully represents the quantum policy for deployment purposes.
    All real-time control uses a simulated backend on Raspberry Pi, not a real QPU (Sec III-A, Sec V-b).
  • domain assumption The reward function is differentiable with respect to policy parameters.
    End-to-end gradient training through model-based rollouts assumes differentiability, but the reward includes a conditional term only when the pole angle is within +/-15 degrees, which is piecewise rather than globally smooth (Sec III-B-c).
  • domain assumption The 64-value state representation (eight time steps) is a sufficient Markovian summary for control.
    The policy input uses eight components including three delayed actions; no verification of sufficiency is given (Sec IV-A).
  • ad hoc to paper Real-time control requires inference below 15 ms.
    The paper states this threshold for the control loop, but it is not derived from cart-pole dynamics or hardware constraints (Sec V-b).

how reviews work

0 comments
Cite this review

Pith. "Pith review of First Experience with Real-Time Control Using Simulated VQC-Based Quantum Policies." pith.science (2026). https://pith.science/paper/IRSPMM5F

@misc{pith2026250801690,
  author       = {Pith},
  title        = {Pith review of: First Experience with Real-Time Control Using Simulated VQC-Based Quantum Policies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRSPMM5F}},
  note         = {Machine review of arXiv:2508.01690}
}
read the original abstract

This paper investigates the integration of quantum computing into offline reinforcement learning and the deployment of the resulting quantum policy in a real-time control hardware realization of the cart-pole system. Variational Quantum Circuits (VQCs) are used to represent the policy. Classical model-based offline policy search was applied, in which a pure VQC with trainable input-output weights is used as a policy network instead of a classical multilayer perceptron. The goal is to evaluate the potential of deploying quantum architectures in real-world industrial control problems. The experimental results show that the investigated model-based offline policy search is able to generate quantum policies that can balance the hardware cart-pole. A latency analysis reveals that while local simulated execution meets real-time requirements, cloud-based quantum processing remains too slow for closed-loop control.

Figures

Figures reproduced from arXiv: 2508.01690 by the authors.

Figure 1
Figure 1. Classical inputs are encoded via trainable [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Offline cart-pole dataset with multiple time-series plots depicting key [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. The real cart-pole system consists of a cart moving along a track, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Comparison of distribution of average steps per start position slot [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Mean absolute weight magnitudes across checkpoints for Pure VQC [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 18 canonical work pages

  1. [1]

    Quantum cryptogra- phy: Public key distribution and coin tossing,

    C. H. Bennett and G. Brassard, “Quantum cryptogra- phy: Public key distribution and coin tossing,” Theoret- ical Computer Science , vol. 560, pp. 7–11, Dec. 2014, ISSN : 0304-3975. DOI: 10.1016/j.tcs.2014.05.025

    doi:10.1016/j.tcs.2014.05.025 2014

  2. [2]

    Simulated quantum computation of molecular energies,

    A. Aspuru-Guzik, A. D. Dutoi, P. J. Love, and M. Head- Gordon, “Simulated quantum computation of molecular energies,” Science, vol. 309, no. 5741, pp. 1704–1707, 2005

    work page 2005

  3. [3]

    Quantum supremacy through the quantum approximate optimization algo- rithm,

    E. Farhi and A. W. Harrow, “Quantum supremacy through the quantum approximate optimization algo- rithm,” arXiv preprint, 2019. DOI: 10.48550/arXiv.1602. 07674

    work page doi:10.48550/arxiv.1602 2019

  4. [4]

    Cost function dependent barren plateaus in shallow parametrized quantum circuits,

    M. Cerezo, A. Sone, T. V olkoff, L. Cincio, and P. J. Coles, “Cost function dependent barren plateaus in shallow parametrized quantum circuits,” Nature Com- munications, vol. 12, no. 1, Mar. 2021, ISSN : 2041-

    work page 2021

  5. [5]

    Reinforcement learning in robotics: A survey,

    J. Kober, J. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Jour- nal of Robotics Research , vol. 32, pp. 1238–1274, Sep

    work page

  6. [6]

    C. Yu, J. Liu, and S. Nemati, Reinforcement learning in healthcare: A survey , 2020. arXiv: 1908 . 08796 [cs.LG]

    work page 2020

  7. [7]

    Deep reinforcement learning in smart manufacturing: A re- view and prospects,

    C. Li, P. Zheng, Y . Yin, B. Wang, and L. Wang, “Deep reinforcement learning in smart manufacturing: A re- view and prospects,” CIRP Journal of Manufacturing Science and Technology , vol. 40, pp. 75–101, 2023, ISSN : 1755-5817. DOI: https://doi.org/10.1016/j.cirpj. 2022.11.003

    doi:10.1016/j.cirpj 2023

  8. [8]

    Quantum- enhanced machine learning,

    V . Dunjko, J. M. Taylor, and H. J. Briegel, “Quantum- enhanced machine learning,” Physical Review Letters , vol. 117, no. 13, Sep. 2016, ISSN : 1079-7114. DOI: 10. 1103/physrevlett.117.130501

    work page 2016

Show all 25 references
  1. [9]

    S. Y .-C. Chen, C.-H. H. Yang, J. Qi, P.-Y . Chen, X. Ma, and H.-S. Goan, Variational quantum circuits for deep reinforcement learning , 2020. arXiv: 1907.00397 [cs.LG]

    2020 arXiv

  2. [10]

    K ¨olle et al

    M. K ¨olle et al. , Quantum advantage actor-critic for reinforcement learning , 2024. arXiv: 2401 . 07043 [quant-ph]

    2024

  3. [11]

    A study on optimization techniques for variational quantum circuits in reinforcement learning,

    M. K ¨olle, T. Witter, T. Rohe, G. Stenzel, P. Altmann, and T. Gabor, “A study on optimization techniques for variational quantum circuits in reinforcement learning,” in 2024 IEEE International Conference on Quantum Software (QSW), IEEE, 2024, pp. 157–167

    2024

  4. [12]

    Overcoming model bias for robust offline deep reinforcement learn- ing,

    P. Swazinna, S. Udluft, and T. Runkler, “Overcoming model bias for robust offline deep reinforcement learn- ing,” Engineering Applications of Artificial Intelligence, vol. 104, p. 104 366, 2021

    2021

  5. [13]

    Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning,

    A. Skolik, S. Jerbi, and V . Dunjko, “Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning,” Quantum, vol. 6, p. 720, 2022. DOI: 10. 22331/q-2022-05-24-720

    2022

  6. [14]

    Circuit-centric quantum classifiers,

    M. Schuld, A. Bocharov, K. M. Svore, and N. Wiebe, “Circuit-centric quantum classifiers,” Physical Review A, vol. 101, no. 3, 2020, ISSN : 2469-9934. DOI: 10 . 1103/physreva.101.032308

    2020

  7. [15]

    Inter- pretable control by reinforcement learning,

    D. Hein, S. Limmer, and T. A. Runkler, “Inter- pretable control by reinforcement learning,” IFAC- PapersOnLine, vol. 53, no. 2, pp. 8082–8089, 2020

    2020

  8. [16]

    Akiba, S

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A next-generation hyperparameter optimiza- tion framework, 2019. arXiv: 1907.10902 [cs.LG]

    2019 arXiv

  9. [17]

    D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, 2017. arXiv: 1412.6980 [cs.LG]

    2017 arXiv

  10. [18]

    Tensorcircuit: A quantum software framework for the nisq era,

    S.-X. Zhang et al., “Tensorcircuit: A quantum software framework for the nisq era,” Quantum, vol. 7, p. 912, 2023

    2023

  11. [19]

    Differentiable quantum architecture search for job shop scheduling problem,

    Y . Sun, J. Liu, Y . Ma, and V . Tresp, “Differentiable quantum architecture search for job shop scheduling problem,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2024, pp. 236–240

    2024

  12. [20]

    S. Y .-C. Chen, Differentiable quantum architecture search in asynchronous quantum reinforcement learn- ing, 2024. arXiv: 2407.18202 [quant-ph]

    2024 arXiv

  13. [21]

    Model-based offline quantum reinforcement learning,

    S. Eisenmann, D. Hein, S. Udluft, and T. A. Runkler, “Model-based offline quantum reinforcement learning,” in 2024 IEEE International Conference on Quantum Computing and Engineering (QCE) , IEEE, Sep. 2024, pp. 1490–1496. DOI: 10.1109/qce60285.2024.00175

    2024

  14. [22]

    Schuld and F

    M. Schuld and F. Petruccione, Supervised Learning with Quantum Computers , 1st. Springer Publishing Com- pany, Incorporated, 2018, ISBN : 3319964232. DOI: 10. 1007/978-3-319-96424-9

    2018

  15. [23]

    Matrix product state representations,

    D. Perez-Garcia, F. Verstraete, M. M. Wolf, and J. I. Cirac, “Matrix product state representations,” arXiv preprint, 2007. DOI: 10.48550/arXiv.0608197

    2007 doi

  16. [1723]

    DOI: 10.1038/s41467-021-21728-w

    doi

  17. [2013]

    DOI: 10.1177/0278364913495721

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.