Pith. sign in

REVIEW 5 major objections 4 minor 11 references

Reinforcement Learning for Quantum Circuit Design: Using Matrix Representations

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reinforcement learning, using matrix or tensor-network representations of quantum circuits, can automatically discover target circuits such as Bell states, GHZ, and Toffoli from a universal gate set.

desk verdict Solution-primed rewards and a no-op DQN update sink this clearly written but circular small-circuit RL paper. read the letter →

arxiv 2501.16509 v1 pith:PE7JGKKX submitted 2025-01-27 quant-ph cs.AI

Zhiyuan Wang , Chunlin Feng , Christopher Poon , Lijian Huang , Xingjian Zhao , Yao Ma , Tianfan Fu , Xiao-Yang Liu This is my paper · ORCID
classification quant-phcs.AI
keywords quantumcircuitdesignreinforcementlearningQ-learningdeepQ-networkMarkovdecisionprocessmatrixrepresentationtensornetworkToffoligate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that reinforcement learning can automate a piece of circuit design that is normally done by hand: searching a small universal gate set for a sequence that implements a target quantum operation. To make the search tractable, it casts the synthesis problem as a Markov decision process in three equivalent-looking ways — forward matrix multiplication, reverse matrix multiplication back to the identity, and tensor-network evolution of a state vector — and runs Q-learning and Deep Q-Network agents on ten standard target circuits: four Bell states, SWAP, iSWAP, CZ, GHZ, Z, and Toffoli. The paper reports that both algorithms find the target circuits on the simpler tasks, with the tensor-network and reverse-matrix variants succeeding most often, and concludes that the approach is an automatic, scalable alternative to hand-crafted heuristics. A sympathetic reader would take the contribution to be a proof-of-concept that RL, with the right state space and a correct MDP, can synthesize these circuits rather than a demonstration of open-ended search.

What carries the argument

The object that carries the argument is the state tree of unitary matrices. Starting from the identity, each allowed gate left-multiplies the current matrix, so the agent's search space is a rooted tree with branching factor equal to the number of gates in the action set and depth equal to the maximum allowed circuit length; the size of this tree, $c^0 + c^1 + \cdots + c^b$, is the paper's measure of task difficulty. The reward function is the sparse signal: a single positive reward of 100 at the exact state-action pair that produces the target unitary (or target state), with all other transitions receiving 0. The same tree construction, run with inverse gates or with tensor-network states, produces the two alternative formulations; the tensor-network version can also apply two-gate blocks as one action.

What would settle it

Run the matrix-representation experiment with the reward changed to reward any sequence whose final unitary matches the target to fidelity >0.99, and run the Toffoli experiment with no expert trajectory in the replay buffer; if success on Bell states and Toffoli collapses, the reported 'discovery' is imitation of the reward's built-in answer.

Watch

Extended reading notes

Core claim

The paper's central claim is that a generic MDP over gate matrices lets an off-the-shelf Q-learning or DQN agent rediscover known quantum circuits from a bare gate set. In the forward matrix representation the state is the current cumulative unitary $S$, the action is a gate matrix $A$, and the transition is $S' = A S$; the reward is 100 only for the exact state-action pair that lands on the target unitary, and 0 otherwise. In the reverse representation the agent starts from the target unitary, applies inverse gates until the identity is reached, and then reverses the order. In the tensor-network representation the state is the quantum state vector itself, and an action may apply a block of gates at once. The paper evaluates success over 100 training rounds per task, reporting, for example, 86–100% success on Bell state $|\Phi^+\rangle$ and 87–91% on the Toffoli gate when an expert trajectory is seeded, while harder tasks such as iSWAP and $|\Psi^-\rangle$ stay below 10%.

Load-bearing premise

The approach only counts as automatic design if the sparse reward is a genuine success signal — rather than a pre-specified answer — and if the Toffoli expert trajectory is helpful guidance rather than a memorized solution.

Editorial extensions

If this is right

  • If the MDP formulations work on these ten tasks, the same scaffolding can be applied to larger gate sets and longer circuit lengths without redesigning the search.
  • The reported success ratios give a concrete ranking of representations: tensor-network Q-learning dominates on the Bell states, while reverse-matrix Q-learning is strongest on the Toffoli task.
  • The state-tree size formula provides a quantitative difficulty measure that can be used to predict which synthesis tasks are within reach of tabular Q-learning.
  • On the iSWAP task the discovered circuit differs from Qiskit's library version, showing the search can return non-identical solutions that still match the target unitary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the matrix-representation reward fires only at the exact known state-action pair, a fair test of 'discovery' would require a reward that is sparse but not solution-specific, e.g., any terminal unitary within fidelity threshold; the current design makes the task a path-following exercise rather than a search.
  • The state-tree size formula implies a natural curriculum: start agents on low-branching, shallow tasks (like Bell $|\Phi^+\rangle$) and increase branching and depth only after convergence, which is a testable way to extend the method beyond the ten circuits reported.
  • The success-ratio table suggests that DQN underperforms tabular Q-learning on small state spaces, so the deep-network advantage would only show on tasks large enough to make the table impractical — a claim the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes three MDP formulations for quantum circuit design (matrix representation, reverse matrix representation, and tensor-network representation) and applies Q-learning and DQN to synthesize a set of textbook quantum circuits: four Bell states, SWAP, iSWAP, CZ, GHZ, Z, and Toffoli. The authors claim that RL provides an automatic and scalable approach to quantum circuit design, and they report success ratios in Table 6 for Q-learning, DQN, and Q-learning with the tensor-network representation.

Significance. If the central claim were supported, a demonstration that generic Q-learning and DQN can discover quantum circuits from a target specification would be a useful incremental result, though not surprising given prior work on RL for circuit synthesis. The paper provides formal MDP definitions and a comparative empirical study across ten circuits, which is a reasonable structure. However, the significance is sharply reduced by the design of the reward functions and the use of expert trajectories: the rewards are placed on the known solution path, and the Toffoli experiments are seeded with the expert solution. These choices mean the experiments do not actually test automatic circuit discovery. In addition, the DQN target-network update as written is a no-op, so the reported DQN results are not trustworthy. The paper does ship a code repository link, which is a positive, but the code is not described in sufficient detail to verify the experiments.

major comments (5)
  1. [DQN Algorithm] The target network update is written as θ = (1 − α)θ + αθ, which simplifies identically to θ = θ and performs no update. This means the DQN described in the paper has no effective target network and reduces to a plain Q-network with a periodical no-op. Consequently, the DQN results in Table 6 (e.g., Bell |Φ+⟩ DQN 33%, Toffoli DQN 1%) cannot be interpreted as coming from the algorithm claimed. This is a load-bearing error for the DQN part of the central claim.
  2. [Matrix Representation / Reward function] The reward for the Bell-state example is R(s=S1, a=CNOT01)=100, where S1 is the state after applying H0 and CNOT01 is exactly the second gate of the known solution circuit in Fig. 1. For a two-gate circuit, specifying the rewarded penultimate state-action pair is equivalent to specifying the entire circuit. The same solution-priming appears in the Reverse Matrix Representation (R(S−1_5, H−1_0)=100) and the TN Representation (R(S0, (H0,CNOT01))=100). These reward functions do not specify the target state; they place a marker on the known solution trajectory. Therefore, the reported success in Table 6 for Bell states does not demonstrate automatic circuit design. Moreover, the appendix 'Reward Calculation' says that 'the reward for all tasks' is a generic fidelity-based reward comparing the current unitary with the target unitary, which directly contradicts the Section 3 reward definitions. The paper never states which reward function was actually used in the experiments.
  3. [Appendix: Expert Trajectories for Toffoli Gate] The Toffoli experiments explicitly store an expert trajectory in the replay buffer and apply it over multiple iterations (10 iterations for Q-learning, 150 for DQN). The reported Toffoli Q-learning success of 87% is therefore trained on the answer sequence, and the DQN success of 1–3% despite this priming indicates that the DQN implementation as described is not learning effectively. Additionally, the Toffoli action set A = {CNOT21, H0, CP10, CP−1_10, CP20} is hand-chosen to be exactly the gates appearing in the known solution in Fig. 6. The Toffoli result is thus closer to imitation than to automatic synthesis, undermining the claim that RL 'automatically' finds the Toffoli circuit.
  4. [Section 3 vs. Table 4/Table 5] There is an inconsistency between the formal MDP definition and the experimental setup. The Matrix Representation in Section 3 defines the action set as A = {H0, H1, T0, T1, CNOT01}, giving 5 actions and a state tree of 31 states. However, Table 5 lists a 6-action set that includes CNOT10, and Table 4 reports 6 actions and a state-space size of 43 for Bell |Φ+⟩ (consistent with c=6, b=2). This discrepancy means the reported experiments may not correspond to the MDPs formally defined in Section 3, and the paper needs to reconcile these definitions.
  5. [Table 6] Even under the solution-primed reward and expert-trajectory setup, the success ratios in Table 6 are very low for most tasks: iSWAP 2% (Q-learning), SWAP 10%, Bell |Ψ−⟩ 5%, DQN Toffoli 1%, and TN SWAP 3%. These numbers do not support the abstract's claim of an 'automatic and scalable' approach; for several tasks the method essentially fails. The paper's conclusion that 'both Q-learning and DQN algorithms could find the target quantum circuits' is therefore an overstatement relative to the evidence.
minor comments (4)
  1. [Q-Learning Algorithm] The learning parameters stated in the Q-learning section are α = 0.5, γ = 0.9, ε = 0.2, while the appendix 'Q-learning and DQN Environment' lists α = 0.1, γ = 0.95, and ε decaying from 1.0. The paper should clarify which parameter set was used for the reported results.
  2. [Introduction] The phrase 'universal gate set G = {H, T, CNOT}' is used loosely: the Toffoli experiments use controlled-phase gates not in this set, and the Bell-state experiments use X, Z, and S gates. The paper should either use a precisely defined gate set or explain the deviations.
  3. [Matrix Representation] The claimed advantage that the matrix representation reduces the state space because different gate sequences can yield the same matrix is not realized in the formal definition, which describes a tree with 31 distinct nodes for two levels and does not merge equivalent states. The authors should clarify whether duplicate matrices are merged and how the state-space sizes in Table 4 were computed.
  4. [Appendix: Q-learning and DQN Environment] The code is provided as a link to a GitHub repository under a course project directory. For reproducibility, the paper should provide a stable artifact, ideally with a DOI or a tagged release, and specify the exact versions of Qiskit and gym used.

Circularity Check

4 steps flagged · score 8.0 of 10

The reported RL successes are encoded in the reward and replay definitions: the Bell-state MDPs reward the exact known solution action (or composite action), and the Toffoli experiments seed the replay buffer with the expert trajectory.

  1. self definitional [Section 'Matrix Representation', Reward function (after Eq. 3)]
    "Reward function R: At state S1, taking action a = CNOT01, the reward is R(s = S1, a= CNOT01) = 100; otherwise, R(s, a) = 0."

    S1 is defined by Eq. (4) as the state obtained by applying H0 to I4, and S10 = CNOT01·S1 is the target U in Eq. (5). Thus the nonzero reward is placed on the exact penultimate (state, action) pair of the known two-gate Bell solution. The learned Q-table entry Q(S1,CNOT01)=100 is a copy of this reward definition, not a discovered circuit. Specifying this reward is equivalent to specifying the whole circuit, so the 'finding' reduces by construction to the input reward function.

  2. self definitional [Section 'Reverse Matrix Representation', Reward function (after Eq. 6)]
    "At state S−1 5, taking action a = H −1 0, the reward R(s = S−1 5, a= H −1 0) = 100 ; otherwise, R(s, a) = 0."

    Equation (7) defines S−1_5 as CNOT^{-1}_{01}·S^{-1}_0, and Eq. (8) shows that H^{-1}_0 applied there yields I4, the terminal state. The rewarded state-action pair is therefore exactly the inverse of the known Bell circuit (CNOT then H). The reported reverse-representation success is forced by the reward definition rather than by autonomous search over alternative decompositions.

2 more flagged steps
  1. self definitional [Section 'Tensor Network Representation', Reward function (after Eq. 9)]
    "Taking action (H0, CNOT01) results in the TN representation in Fig. 4. ... Reward function R: At state S0, taking action a = ( H0, CNOT01), the reward R(s = S0, a = (H0, CNOT01)) = 100; otherwise, R(s, a) = 0."

    The composite action (H0, CNOT01) is the entire target circuit of Fig. 1 packaged as a single action, and the only nonzero reward is given for choosing exactly that action from S0. Example Eq. (10) simply applies that action to |00> and obtains |Φ+>. With the answer available as one action and rewarded with 100, the reported 100% TN success is definitional, not a search result.

  2. fitted input called prediction [Section 'Experiment Results', Toffoli Gate; Appendix 'Expert Trajectories for Toffoli Gate']
    "An expert trajectory were stored in the replay buffer to improve learning efficiency. For Matrix Representation, the expert trajectory is: {H0 → CP10 → CNOT21 → CP−1 10 → CNOT21 → CP20 → H0}. ... Both Q-Learning and DQN use an expert action sequence to embed optimal behavior for constructing a Toffoli gate."

    The expert trajectory is the known target circuit from Fig. 6. The appendix states it is 'applied over multiple iterations' (10 for Q-learning, 150 for DQN) and that transitions from it are replayed/stored. The reported Toffoli success ratios (87% Q-learning, 91% reverse) therefore measure how well the agent imitates the inserted answer sequence, not whether RL autonomously designed the Toffoli circuit. The claimed 'automatic design' result is an evaluation of the provided expert data.

full rationale

The central demonstration of automatic quantum circuit design is self-defeating because the reward and replay specifications already contain the circuits that are reported as discovered. For the Bell-state running example, each of the three MDPs places its sole +100 reward on the exact known solution: (S1, CNOT01) in the matrix representation, (S^{-1}_5, H^{-1}_0) in the reverse representation, and the composite action (H0, CNOT01) from S0 in the tensor-network representation. Since S1 and S^{-1}_5 are defined by the first gate of the known decomposition, these rewards encode the complete two-gate circuit; the resulting Q-tables (Tables 1-3) are the reward function copied into the table. The Toffoli experiment is more explicit: an expert trajectory equal to the target decomposition is stored in the replay buffer and applied over many iterations, so the 87%/91% success ratios are imitation scores. The appendix 'Reward Calculation' proposes a generic unitary-fidelity reward, which would be a legitimate goal specification, but it contradicts the Section 3 reward definitions and does not repair the reported Bell-state and Toffoli results. Because the paper's flagship examples reduce by construction to hand-placed rewards and seeded expert trajectories, the abstract's claim of an 'automatic and scalable approach' is not supported by the experiments as described. No self-citation circularity is present; the circularity is internal to the MDP and replay-buffer definitions.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-designed rewards and action sets, plus standard RL assumptions, rather than on a principled generative model.

free parameters (5)
  • Reward value R=100 = 100
    Chosen by hand; all tasks use the same reward for reaching the target, which sets the scale for Q-values.
  • Fidelity threshold 0.99 = 0.99
    Defines success in the reward calculation; a lower threshold would make tasks easier.
  • Q-learning hyperparameters = alpha=0.5 or 0.1, gamma=0.9 or 0.95, epsilon=0.2 or decaying
    Stated in the Q-learning and DQN sections; no sensitivity analysis is provided.
  • DQN architecture = 3 fully connected layers, 128 neurons each
    One architecture is used for all tasks; no tuning or comparison is reported.
  • Expert trajectory for Toffoli = H0 -> CP10 -> CNOT21 -> CP10^-1 -> CNOT21 -> CP20 -> H0
    The solution sequence is provided to the agent; this structural choice bypasses search and biases the reported success.
assumptions (4)
  • domain assumption Unitary fidelity |Tr(S'†U)|/2^n > 0.99 is a valid measure of circuit correctness.
    Used in the reward calculation; may not distinguish global phases or require exact decomposition.
  • domain assumption The finite tree state space with depth up to 7 is exhaustive for the considered tasks.
    The action sets are assumed to contain the gates needed to build each target circuit; no proof is given that the gate sets are sufficient or that the depth bounds are tight.
  • standard math Q-learning converges to the optimal Q-values for these finite MDPs under tabular updates.
    Assumed as background; convergence requires visiting all state-action pairs infinitely often, which may not hold in limited episodes.
  • ad hoc to paper The reward function R(s1, CNOT01)=100 and the expert trajectory do not bias the evaluation.
    The main-text reward is defined on the known solution path, and the Toffoli agent is given the solution, so the evaluation measures imitation rather than discovery.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning for Quantum Circuit Design: Using Matrix Representations." pith.science (2026). https://pith.science/paper/PE7JGKKX

@misc{pith2026250116509,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning for Quantum Circuit Design: Using Matrix Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PE7JGKKX}},
  note         = {Machine review of arXiv:2501.16509}
}
read the original abstract

Quantum computing promises advantages over classical computing. The manufacturing of quantum hardware is in the infancy stage, called the Noisy Intermediate-Scale Quantum (NISQ) era. A major challenge is automated quantum circuit design that map a quantum circuit to gates in a universal gate set. In this paper, we present a generic MDP modeling and employ Q-learning and DQN algorithms for quantum circuit design. By leveraging the power of deep reinforcement learning, we aim to provide an automatic and scalable approach over traditional hand-crafted heuristic methods.

Figures

Figures reproduced from arXiv: 2501.16509 by the authors.

Figure 1
Figure 1. A quantum circuit to generate Bell state [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. State tree in reverse matrix representation for search [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. TN representation of Fig. 1 [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: State tree in TN representation for searching the [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: A design of Toffoli Gate with fewer gates. [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: A quantum circuit to generate Bell state [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 10
Figure 10. Figure 10: A quantum circuit to implement the SWAP gate [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 11
Figure 11. Figure 11: A quantum circuit to implement the iSWAP gate [PITH_FULL_IMAGE:figures/full_fig_p006_11.png]
Figure 12
Figure 12. Figure 12: A quantum circuit to implement the iSWAP gate [PITH_FULL_IMAGE:figures/full_fig_p007_12.png]
Figure 13
Figure 13. Figure 13: A quantum circuit to implement the CZ gate and [PITH_FULL_IMAGE:figures/full_fig_p007_13.png]
Figure 14
Figure 14. Figure 14: A quantum circuit to generate a GHZ state. [PITH_FULL_IMAGE:figures/full_fig_p007_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 7 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    B.; Hirayama, T.; Yamanaka, K.; and Nishitani, Y

    Ali, M. B.; Hirayama, T.; Yamanaka, K.; and Nishitani, Y. 2015. Quantum cost reduction of reversible circuits using new Toffoli decomposition techniques. In International Conference on Computational Science and Computational Intelligence (CSCI), 59--64. IEEE

    work page 2015

  4. [4]

    C.; Barends, R.; Biswas, R.; Boixo, S.; Brandao, F

    Arute, F.; Arya, K.; Babbush, R.; Bacon, D.; Bardin, J. C.; Barends, R.; Biswas, R.; Boixo, S.; Brandao, F. G.; Buell, D. A.; et al. 2019. Quantum supremacy using a programmable superconducting processor. Nature, 574(7779): 505--510

    work page 2019

  5. [5]

    A.; Khanday, F

    Bhat, H. A.; Khanday, F. A.; and Shah, K. A. 2022. Optimal quantum circuit decomposition of reversible gates on IBM quantum computer. In International Conference on Multimedia, Signal Processing and Communication Technologies (IMPACT), 1--4. IEEE

    work page 2022

  6. [6]

    S.; Kumar, A.; Singh, H.; Singh, M.; Kaur, K.; Usman, M.; and Buyya, R

    Gill, S. S.; Kumar, A.; Singh, H.; Singh, M.; Kaur, K.; Usman, M.; and Buyya, R. 2021. Quantum Computing: A Taxonomy, Systematic Review and Future Directions. arXiv:2010.15559

    arXiv 2021

  7. [7]

    Liu, X.-Y.; and Zhang, Z. 2023. Classical Simulation of Quantum Circuits: Parallel Environments and Benchmark. In Advances in Neural Information Processing Systems, volume 36, 67082--67102

    work page 2023

  8. [8]

    Meirom, E.; Maron, H.; Mannor, S.; and Chechik, G. 2022. Optimizing tensor network contraction using reinforcement learning. In International Conference on Machine Learning, 15278--15292. PMLR

    work page 2022

Show all 11 references
  1. [9]

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602

    2013 arXiv

  2. [10]

    Sogabe, T.; Kimura, T.; Chen, C.-C.; Shiba, K.; Kasahara, N.; Sogabe, M.; and Sakamoto, K. 2022. Model-free deep recurrent Q-network reinforcement learning for quantum circuit architectures design. Quantum Reports, 4(4): 380--389

    2022

  3. [11]

    J.; and Dayan, P

    Watkins, C. J.; and Dayan, P. 1992. Q-learning. Machine Learning, 8(3-4): 279--292

    1992

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.