{"id":"6618be58-4ca2-4101-ad2d-4ea6b9e5de65","arxiv_id":"2501.16509","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper applies Q-learning and DQN to synthesizing small quantum circuits via matrix-based MDPs, reporting success rates from 100% down to 1% depending on task and algorithm.","lead":"The paper applies Q-learning and DQN to synthesizing small quantum circuits (Bell states, SWAP, CZ, GHZ, Toffoli) using matrix-based MDP models. It reports mixed success rates and relies on hand-crafted rewards and expert trajectories for several tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported successes are not evidence of automatic design: the running-example rewards are placed at the exact known solution action, and the Toffoli experiments seed the replay buffer with the expert trajectory.","rationale":"The paper's own formal definitions place the only positive reward on the exact state-action pair that completes the known solution, which is a form of solution-priming. The Toffoli experiments explicitly seed the replay buffer with the expert trajectory, so the agent is not discovering the circuit from a target specification alone. This directly attacks the central 'automatic' claim: if the reward or replay already contains the answer, then high success rates only measure imitation and credit assignment, not design. The inconsistency between the hand-coded rewards in Section 3 and the generic unitary-fidelity reward in the Appendix further weakens the evidence, because it is unclear which reward function actually produced Table 6. A simple re-run without the solution-specific reward and without expert demonstrations would settle whether the method can discover circuits from a generic target specification. Since this concern matches the reader's weakest assumption and reinforces the rejection, no change to the reader's verdict is needed.","tokens_in":10080,"tokens_out":5079,"duration_ms":51563,"concrete_test":"Re-run the Bell-state Matrix Representation task with the generic reward from the Appendix (reward 100 if |Tr(S'† U)|/2^n > 0.99) instead of the hand-coded R(S1, CNOT01)=100, and re-run the Toffoli DQN task with an empty replay buffer (no expert trajectories). If Bell success remains high and Toffoli remains above chance, automaticity is supported; if Bell collapses or Toffoli DQN drops to near zero, the published success ratios depend on solution-specific reward and expert seeding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Q-learning and DQN 'automatically' design quantum circuits is undermined by the reward and replay definitions. In the Matrix Representation, the only nonzero reward is R(s=S1, a=CNOT01)=100, where S1 is the state after applying H0 to the identity and CNOT01 is exactly the second gate of the known Bell circuit. The Reverse and TN representations do the same: R(S−1_5, H−1_0)=100 and R(S0, (H0, CNOT01))=100. These are not generic target-specification rewards; they are hand-placed markers on the known solution trajectory. For a two-gate circuit, specifying the rewarded penultimate state-action pair is equivalent to specifying the entire circuit. For Toffoli, the paper states that an expert trajectory is stored in the replay buffer and applied over multiple iterations, so the reported Toffoli Q-learning success of 87% is trained on the answer. The later 'Reward Calculation' appendix describes a generic unitary-fidelity reward for 'all tasks', which directly contradicts the Section 3 definitions. Either the formal MDPs are not what was evaluated, or the evaluation was solution-primed; neither supports the abstract's automatic-design claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three MDP formulations for quantum circuit design (matrix representation, reverse matrix representation, and tensor-network representation) and applies Q-learning and DQN to synthesize a set of textbook quantum circuits: four Bell states, SWAP, iSWAP, CZ, GHZ, Z, and Toffoli. The authors claim that RL provides an automatic and scalable approach to quantum circuit design, and they report success ratios in Table 6 for Q-learning, DQN, and Q-learning with the tensor-network representation.","tokens_in":10320,"tokens_out":5483,"duration_ms":49970,"significance":"If the central claim were supported, a demonstration that generic Q-learning and DQN can discover quantum circuits from a target specification would be a useful incremental result, though not surprising given prior work on RL for circuit synthesis. The paper provides formal MDP definitions and a comparative empirical study across ten circuits, which is a reasonable structure. However, the significance is sharply reduced by the design of the reward functions and the use of expert trajectories: the rewards are placed on the known solution path, and the Toffoli experiments are seeded with the expert solution. These choices mean the experiments do not actually test automatic circuit discovery. In addition, the DQN target-network update as written is a no-op, so the reported DQN results are not trustworthy. The paper does ship a code repository link, which is a positive, but the code is not described in sufficient detail to verify the experiments.","major_comments":[{"comment":"The target network update is written as θ = (1 − α)θ + αθ, which simplifies identically to θ = θ and performs no update. This means the DQN described in the paper has no effective target network and reduces to a plain Q-network with a periodical no-op. Consequently, the DQN results in Table 6 (e.g., Bell |Φ+⟩ DQN 33%, Toffoli DQN 1%) cannot be interpreted as coming from the algorithm claimed. This is a load-bearing error for the DQN part of the central claim.","section":"DQN Algorithm"},{"comment":"The reward for the Bell-state example is R(s=S1, a=CNOT01)=100, where S1 is the state after applying H0 and CNOT01 is exactly the second gate of the known solution circuit in Fig. 1. For a two-gate circuit, specifying the rewarded penultimate state-action pair is equivalent to specifying the entire circuit. The same solution-priming appears in the Reverse Matrix Representation (R(S−1_5, H−1_0)=100) and the TN Representation (R(S0, (H0,CNOT01))=100). These reward functions do not specify the target state; they place a marker on the known solution trajectory. Therefore, the reported success in Table 6 for Bell states does not demonstrate automatic circuit design. Moreover, the appendix 'Reward Calculation' says that 'the reward for all tasks' is a generic fidelity-based reward comparing the current unitary with the target unitary, which directly contradicts the Section 3 reward definitions. The paper never states which reward function was actually used in the experiments.","section":"Matrix Representation / Reward function"},{"comment":"The Toffoli experiments explicitly store an expert trajectory in the replay buffer and apply it over multiple iterations (10 iterations for Q-learning, 150 for DQN). The reported Toffoli Q-learning success of 87% is therefore trained on the answer sequence, and the DQN success of 1–3% despite this priming indicates that the DQN implementation as described is not learning effectively. Additionally, the Toffoli action set A = {CNOT21, H0, CP10, CP−1_10, CP20} is hand-chosen to be exactly the gates appearing in the known solution in Fig. 6. The Toffoli result is thus closer to imitation than to automatic synthesis, undermining the claim that RL 'automatically' finds the Toffoli circuit.","section":"Appendix: Expert Trajectories for Toffoli Gate"},{"comment":"There is an inconsistency between the formal MDP definition and the experimental setup. The Matrix Representation in Section 3 defines the action set as A = {H0, H1, T0, T1, CNOT01}, giving 5 actions and a state tree of 31 states. However, Table 5 lists a 6-action set that includes CNOT10, and Table 4 reports 6 actions and a state-space size of 43 for Bell |Φ+⟩ (consistent with c=6, b=2). This discrepancy means the reported experiments may not correspond to the MDPs formally defined in Section 3, and the paper needs to reconcile these definitions.","section":"Section 3 vs. Table 4/Table 5"},{"comment":"Even under the solution-primed reward and expert-trajectory setup, the success ratios in Table 6 are very low for most tasks: iSWAP 2% (Q-learning), SWAP 10%, Bell |Ψ−⟩ 5%, DQN Toffoli 1%, and TN SWAP 3%. These numbers do not support the abstract's claim of an 'automatic and scalable' approach; for several tasks the method essentially fails. The paper's conclusion that 'both Q-learning and DQN algorithms could find the target quantum circuits' is therefore an overstatement relative to the evidence.","section":"Table 6"}],"minor_comments":[{"comment":"The learning parameters stated in the Q-learning section are α = 0.5, γ = 0.9, ε = 0.2, while the appendix 'Q-learning and DQN Environment' lists α = 0.1, γ = 0.95, and ε decaying from 1.0. The paper should clarify which parameter set was used for the reported results.","section":"Q-Learning Algorithm"},{"comment":"The phrase 'universal gate set G = {H, T, CNOT}' is used loosely: the Toffoli experiments use controlled-phase gates not in this set, and the Bell-state experiments use X, Z, and S gates. The paper should either use a precisely defined gate set or explain the deviations.","section":"Introduction"},{"comment":"The claimed advantage that the matrix representation reduces the state space because different gate sequences can yield the same matrix is not realized in the formal definition, which describes a tree with 31 distinct nodes for two levels and does not merge equivalent states. The authors should clarify whether duplicate matrices are merged and how the state-space sizes in Table 4 were computed.","section":"Matrix Representation"},{"comment":"The code is provided as a link to a GitHub repository under a course project directory. For reproducibility, the paper should provide a stable artifact, ideally with a DOI or a tagged release, and specify the exact versions of Qiskit and gym used.","section":"Appendix: Q-learning and DQN Environment"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft with several internal inconsistencies between the formal MDP definitions, the appendix reward description, and the experimental setup. The most serious issue is that the reward functions and the Toffoli expert trajectory encode the known solution, so the reported 'successes' do not establish automatic circuit design. Even if the authors were to replace the rewards with the fidelity-based reward from the appendix and remove the expert trajectories, the observed success rates in Table 6 would likely drop further, calling into question the central claim. The DQN target-network update is also a no-op as written, which would require a rerun of all DQN experiments. Given the scope of these problems, I do not see a path to acceptance without a fundamental rework of the experiments and claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a good example of why reviewers should check the reward function before believing the headline. The abstract promises an automatic, scalable RL approach to quantum circuit design, but the implemented rewards and replay-buffer seeding already contain the answer. The Matrix Representation gives +100 only for the exact state-action pair (S1, CNOT01) — that is, the second gate of the known Bell circuit. Reverse and TN versions do the same. For Toffoli, the paper says an expert trajectory is stored in the replay buffer and replayed for 10 (Q-learning) or 150 (DQN) iterations. That turns \"discovery\" into imitation. The later appendix describes a generic unitary-fidelity reward for \"all tasks,\" which contradicts the Section 3 definitions; either the MDPs were not what was evaluated, or the evaluation was solution-primed.\n\nThere is also a concrete math error: the DQN target network update is written as θ = (1−α)θ + αθ, which simplifies to θ = θ. That is a no-op. The DQN algorithm as described cannot train a target network, so the DQN results in Table 6 are unexplained. This is load-bearing, not a typo.\n\nWhat the paper does well: it lays out three MDP encodings (matrix, reverse matrix, tensor-network state) clearly, enumerates 10 small circuit tasks with explicit action sets and state-space sizes, and honestly reports low success ratios for harder tasks like iSWAP (2–5%) and Toffoli DQN (1–3%). The composite-action tensor-network representation is a minor but real twist. The code is linked, though it appears to be a course repository.\n\nThe bottom line: the central claim is not supported. The successes on Bell and Toffoli are primed by construction, and the DQN description is internally inconsistent. The paper's own numbers show poor scalability, which undercuts the \"scalable\" promise. A reader interested in RL-for-quantum pitfalls will learn something here, but it is not a reliable method contribution.\n\nRecommendation: desk reject. It would not survive serious review without a complete rewrite of the reward design and DQN section, and even then the contribution would be incremental.","headline":"Solution-primed rewards and a no-op DQN update sink this clearly written but circular small-circuit RL paper.","tokens_in":10892,"tokens_out":3797,"would_cite":false,"duration_ms":31445,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning, using matrix or tensor-network representations of quantum circuits, can automatically discover target circuits such as Bell states, GHZ, and Toffoli from a universal gate set.","keywords":["quantum circuit design","reinforcement learning","Q-learning","deep Q-network","Markov decision process","matrix representation","tensor network","Toffoli gate"],"falsifier":"Run the matrix-representation experiment with the reward changed to reward any sequence whose final unitary matches the target to fidelity >0.99, and run the Toffoli experiment with no expert trajectory in the replay buffer; if success on Bell states and Toffoli collapses, the reported 'discovery' is imitation of the reward's built-in answer.","tokens_in":9839,"feed_emoji":"⚛️","tokens_out":7476,"duration_ms":68691,"temperature":0.7,"pith_summary":"This paper tries to establish that reinforcement learning can automate a piece of circuit design that is normally done by hand: searching a small universal gate set for a sequence that implements a target quantum operation. To make the search tractable, it casts the synthesis problem as a Markov decision process in three equivalent-looking ways — forward matrix multiplication, reverse matrix multiplication back to the identity, and tensor-network evolution of a state vector — and runs Q-learning and Deep Q-Network agents on ten standard target circuits: four Bell states, SWAP, iSWAP, CZ, GHZ, Z, and Toffoli. The paper reports that both algorithms find the target circuits on the simpler tasks, with the tensor-network and reverse-matrix variants succeeding most often, and concludes that the approach is an automatic, scalable alternative to hand-crafted heuristics. A sympathetic reader would take the contribution to be a proof-of-concept that RL, with the right state space and a correct MDP, can synthesize these circuits rather than a demonstration of open-ended search.","feed_headline":"RL agents synthesize quantum circuits from a gate set","feed_subtitle":"Q-learning and DQN find Bell, GHZ, SWAP, and Toffoli circuits in matrix and tensor-network MDPs.","key_machinery":"The object that carries the argument is the state tree of unitary matrices. Starting from the identity, each allowed gate left-multiplies the current matrix, so the agent's search space is a rooted tree with branching factor equal to the number of gates in the action set and depth equal to the maximum allowed circuit length; the size of this tree, $c^0 + c^1 + \\cdots + c^b$, is the paper's measure of task difficulty. The reward function is the sparse signal: a single positive reward of 100 at the exact state-action pair that produces the target unitary (or target state), with all other transitions receiving 0. The same tree construction, run with inverse gates or with tensor-network states, produces the two alternative formulations; the tensor-network version can also apply two-gate blocks as one action.","core_discovery":"The paper's central claim is that a generic MDP over gate matrices lets an off-the-shelf Q-learning or DQN agent rediscover known quantum circuits from a bare gate set. In the forward matrix representation the state is the current cumulative unitary $S$, the action is a gate matrix $A$, and the transition is $S' = A S$; the reward is 100 only for the exact state-action pair that lands on the target unitary, and 0 otherwise. In the reverse representation the agent starts from the target unitary, applies inverse gates until the identity is reached, and then reverses the order. In the tensor-network representation the state is the quantum state vector itself, and an action may apply a block of gates at once. The paper evaluates success over 100 training rounds per task, reporting, for example, 86–100% success on Bell state $|\\Phi^+\\rangle$ and 87–91% on the Toffoli gate when an expert trajectory is seeded, while harder tasks such as iSWAP and $|\\Psi^-\\rangle$ stay below 10%.","pith_inferences":["Because the matrix-representation reward fires only at the exact known state-action pair, a fair test of 'discovery' would require a reward that is sparse but not solution-specific, e.g., any terminal unitary within fidelity threshold; the current design makes the task a path-following exercise rather than a search.","The state-tree size formula implies a natural curriculum: start agents on low-branching, shallow tasks (like Bell $|\\Phi^+\\rangle$) and increase branching and depth only after convergence, which is a testable way to extend the method beyond the ten circuits reported.","The success-ratio table suggests that DQN underperforms tabular Q-learning on small state spaces, so the deep-network advantage would only show on tasks large enough to make the table impractical — a claim the paper does not test."],"forward_implications":["If the MDP formulations work on these ten tasks, the same scaffolding can be applied to larger gate sets and longer circuit lengths without redesigning the search.","The reported success ratios give a concrete ranking of representations: tensor-network Q-learning dominates on the Bell states, while reverse-matrix Q-learning is strongest on the Toffoli task.","The state-tree size formula provides a quantitative difficulty measure that can be used to predict which synthesis tasks are within reach of tabular Q-learning.","On the iSWAP task the discovered circuit differs from Qiskit's library version, showing the search can return non-identical solutions that still match the target unitary."],"supporting_citations":[{"why":"Supplies the Q-learning update rule used to build the Q-tables for the matrix, reverse-matrix, and tensor-network representations.","marker":"Watkins and Dayan 1992"},{"why":"Provides the DQN architecture with replay buffer and target network that the paper adapts to the circuit-design environment.","marker":"Mnih et al. 2013"},{"why":"Prior work applying model-free deep recurrent Q-learning to Bell-GHZ circuit architectures; the paper's baseline and point of comparison.","marker":"Sogabe et al. 2022"},{"why":"Uses reinforcement learning to optimize tensor-network contraction ordering, motivating the tensor-network formulation used here.","marker":"Meirom et al. 2022"},{"why":"The Sycamore circuit is the benchmark that motivates the tensor-network contraction line in the introduction.","marker":"Arute et al. 2019"}],"fun_headline_variants":["RL rediscovers Bell, GHZ, and Toffoli circuits from gate matrices","Matrix MDPs let Q-learning design quantum circuits automatically","RL designs quantum circuits from bare gate sets via DQN","Tensor-network MDPs enable RL to build GHZ and SWAP circuits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach only counts as automatic design if the sparse reward is a genuine success signal — rather than a pre-specified answer — and if the Toffoli expert trajectory is helpful guidance rather than a memorized solution.","fun_headline_variants_meta":{"raw":{"variants":["RL rediscovers Bell, GHZ, and Toffoli circuits from gate matrices","Matrix MDPs let Q-learning design quantum circuits automatically","RL designs quantum circuits from bare gate sets via DQN","Tensor-network MDPs enable RL to build GHZ and SWAP circuits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2866,"prompt_tokens":844,"completion_tokens":2022,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":460,"tokens_out":2022,"duration_ms":14161,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:47:00.198443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the matrix-representation experiment with the reward changed to reward any sequence whose final unitary matches the target to fidelity >0.99, and run the Toffoli experiment with no expert trajectory in the replay buffer; if success on Bell states and Toffoli collapses, the reported 'discovery' is imitation of the reward's built-in answer.","supporting_citations":[{"cited_title":"J.; and Dayan, P","cited_arxiv_id":null,"evidence_quote":"Supplies the Q-learning update rule used to build the Q-tables for the matrix, reverse-matrix, and tensor-network representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work applying model-free deep recurrent Q-learning to Bell-GHZ circuit architectures; the paper's baseline and point of comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Uses reinforcement learning to optimize tensor-network contraction ordering, motivating the tensor-network formulation used here."},{"cited_title":"C.; Barends, R.; Biswas, R.; Boixo, S.; Brandao, F","cited_arxiv_id":null,"evidence_quote":"The Sycamore circuit is the benchmark that motivates the tensor-network contraction line in the introduction."}],"review_version":1}