{"id":"ffca2125-b4ed-4e96-a60a-cda84c03dcbf","arxiv_id":"2508.20377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Noise-aware neural networks can design pulse sequences for qubit state preparation, with DRL better in weak noise and SL better in strong noise.","lead":"The authors train two machine learning agents, deep reinforcement learning and supervised learning, to design control pulses for preparing a qubit state in a noisy environment. They report that supervised learning becomes more competitive when noise is strong, while reinforcement learning wins in weak noise, and both beat a noise-blind baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-Markovian bath memory is absent from the RL state s; the DRL/SL comparison may be an artifact of a restricted observation space.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing issue: the state s in Sec. 2.2 is treated as a sufficient statistic for optimal action selection in a non-Markovian problem, but the auxiliary operators in Eqs. (2)-(3) are not determined by rho alone. The paper's abstract explicitly promises an 'optimal pulse sequence,' and the DRL/SL comparison is meant to inform algorithm choice; if both algorithms are constrained to memoryless feedback on rho, the comparison does not answer the question of which method is better for the true non-Markovian control problem. The proposed check directly tests whether the missing memory variables matter, and it is feasible within the paper's own simulation framework. Missing reward-function details and absent error bars are additional concerns, but they are secondary: the reward omission mainly affects reproducibility, while the state-sufficiency issue affects the interpretation of the headline crossover. I therefore agree with the reader's conditional verdict rather than escalating it.","tokens_in":16430,"tokens_out":10647,"duration_ms":106259,"concrete_test":"Augment the state s with the auxiliary memory operators O_z and O_w (or, equivalently, with the pseudo-mode coordinate of the Lorentzian bath) at each re-encoding step in Fig. 1, retrain both DRL and SL on the same 700/50/242 task split with identical hyperparameters, and compare test-set fidelity at gamma = 1 and gamma = 4. If either algorithm improves by more than the seed-to-seed spread, the rho-only state is insufficient and the reported crossover is an artifact of the restricted observation space rather than a robust property of DRL versus SL.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Fig. 3 is framed as a general statement about DRL versus SL for pulse design in a non-Markovian bosonic environment, but the policy state s in Sec. 2.2 contains only POVM probabilities of the current reduced state, the target state, and the scalar bath parameters (Gamma, gamma, T). The actual dynamics are not closed in the reduced density matrix: Eqs. (2)-(3) introduce auxiliary operators O_z and O_w that carry bath memory and determine future evolution. When the Fig. 1 loop re-encodes only rho_ini = rho_n, the agent is solving a partially observable MDP with a memoryless observation function. Including Gamma, gamma, and T as features does not repair this, because those are fixed per task while O_z and O_w are dynamical functionals of the entire past control history. Thus the 'optimal pulse sequence' claim and the DRL/SL crossover may be specific to this restricted policy class. In the low-gamma colored-noise regime highlighted in Sec. 2.1 (gamma -> 0), memory effects are largest, so the conclusion is least secure precisely where the model is most non-Markovian.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains DRL and SL agents to design piecewise-constant control pulses for arbitrary qubit-to-qubit state preparation in a two-level system coupled to a bosonic bath, using a non-Markovian QSD master equation. Three training strategies are compared: a noise-unaware model (Case 1), models trained for fixed environmental parameters that enter only the simulator (Case 2), and a single model that receives the environmental parameters as input features (Case 3). On a held-out test set of 242 state-preparation tasks, the authors report that noise-aware models outperform the noise-unaware baseline, that the Case 3 network generalizes across environmental parameters, and that DRL is superior in weak-noise environments while SL is more stable and more accurate in strong-noise environments.","tokens_in":16788,"tokens_out":8837,"duration_ms":87843,"significance":"If the central comparison withstands scrutiny, the paper offers useful evidence that environment-aware training can improve quantum control fidelity in a non-Markovian open-system setting, and that a single network with environmental parameters as input features can replace multiple separately trained controllers. The use of a held-out test set, a noise-unaware baseline, and explicit hyperparameter tables are strengths. However, the DRL reward function is never specified, the agent's observation is memoryless despite non-Markovian dynamics, the auxiliary-equation derivation in Appendix A contains apparent inconsistencies, and the main fidelity comparison lacks statistical uncertainty. These gaps leave the central DRL-versus-SL crossover claim only partially supported.","major_comments":[{"comment":"The DRL reward rt is never defined in the text or appendices. Eq. (8) uses rt as the immediate reward in the loss, but there is no statement of how rt is computed from fidelity, step count, or task completion. The reward determines the trained policy and is essential for reproducing the DRL/SL comparison. Please give the exact reward function for all three cases and state how terminal rewards are handled.","section":"Sec. 3.1, Eq. (8)"},{"comment":"The observation s = [P^1_ini, ..., P^4_tar, Gamma, gamma, T] is not a sufficient statistic for optimal action selection in the non-Markovian dynamics used by the simulator. Eqs. (2)-(3) contain auxiliary operators O_z and O_w that are convolutions with bath correlation functions and therefore carry memory of the entire past control history, while the loop in Fig. 1 re-encodes only rho_n. Unless the authors prove that the reduced density matrix plus fixed bath parameters determines the future evolution, the trained policies are restricted to a memoryless policy class. The claim of optimal pulse sequences and the DRL/SL crossover in Fig. 3 may therefore be artifacts of this restricted observation, and the low-gamma regime emphasized in Sec. 2.1 is precisely where memory effects are largest. Please quantify the effect by augmenting s with O_z, O_w, or a finite action history, and re-running the comparison.","section":"Sec. 2.2, Eqs. (2)-(3), Fig. 1"},{"comment":"The derivation of the auxiliary equations is not internally consistent. In Eq. (A.9) the derivative dO_w/dt commutes with O_z rather than O_w, and the subtracted derivative terms are not the appropriate w* derivatives. In addition, Eq. (A.12) claims single-exponential decay for the correlation functions, which is not true for the Bose-Einstein occupation appearing in Eqs. (A.4)-(A.5) for the Lorentz-Drude spectrum at finite temperature. Since Eqs. (2)-(3) are obtained from these equations and drive all reported fidelities, the numerical results need either a corrected derivation or an explicit statement of the single-pole approximation together with an error estimate.","section":"Appendix A, Eqs. (A.8)-(A.9), (A.12)"},{"comment":"Fig. 3 shows only single point estimates of the test-set average fidelity, with no error bars, no number of independent training runs, and no confidence intervals. The DRL/SL differences and the Case (2)/(3) differences in Table 1 are often around 0.01 in fidelity, so the crossover claims require statistical uncertainty estimates. Please add repeated-run means with standard errors or otherwise quantify the variability.","section":"Fig. 3 and Table 1"},{"comment":"The claimed generalization to unseen environmental parameters is not auditable because the training grid for Case (3) is not specified. The text and Fig. 3 distinguish hollow points as unseen parameters, but they do not give the set of (Gamma, gamma, T) combinations used in training. Please state the training grid explicitly and report the held-out points separately, including whether the unseen points lie inside or outside the convex hull of the training parameters.","section":"Sec. 4, Case (3)"}],"minor_comments":[{"comment":"The symbols O_z(t) and O_w(t) are used in Eq. (1) without a non-circular definition; the text says they are convolutions of O_z and O_w, but the reader would benefit from a one-line identification with the noise-averaged operators M[O] and M[Q] from Eq. (A.11).","section":"Sec. 2.1, Eq. (1)"},{"comment":"The epsilon-greedy description is inconsistent with the hyperparameter table: standard DQN uses a decreasing exploration probability epsilon, while the text says epsilon increases from 0 to slightly below 1 as the probability of choosing the currently most advantageous action. Clarify the convention, and state whether validation and test use epsilon=1 for greedy action selection or epsilon=0.","section":"Sec. 3.1 and Table B.2"},{"comment":"The symbol alpha is used for the discount factor in Eq. (5) and for the learning rate in the hyperparameter tables, while gamma is used both for the bath characteristic frequency and for the reward discount factor in Table B.2. Please rename the learning rate and the discount factor to avoid these collisions.","section":"Eqs. (5), (7), and Tables B.2-B.3"},{"comment":"The action-sequence panels do not label the step index on the horizontal axis, and the mapping between the action labels on the vertical axis and the pairs (J, phi) of the action set is not given. Adding this information would make the pulse sequences reproducible.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an incremental extension of the authors' earlier SL work in [46], and the main new content is the DRL comparison and the Case (3) generalization claim. The central comparison is defensible in principle, but the missing reward definition, the memoryless observation issue, and the Appendix A derivation problems require substantial revision. If the authors can provide the corrected derivation, a concrete reward definition, and uncertainty estimates for Fig. 3, the paper could become acceptable; otherwise the DRL/SL crossover claim remains under-supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe thing to know: this paper compares deep RL and supervised learning for pulse design in a noisy two-level system and claims a crossover — DRL wins in weak noise, SL wins in strong noise. That specific crossover is new, but two gaps keep me from trusting it: the DRL reward function is never defined, and the policy state drops the bath memory that the non-Markovian dynamics depend on.\n\nWhat it does well: the evaluation is honest. Held-out test set, noise-unaware baseline for comparison, and a single Case-(3) network generalizes to unseen environment parameters. That last point is the real practical takeaway.\n\nSoft spots, in order:\n\nFirst, the reward. Section 3.1 gives the Q-learning update and the discount factor but never says what r_t is. Is it fidelity, a step penalty, a threshold indicator? That determines the entire policy, so without it the DRL results are not reproducible.\n\nSecond, the state sufficiency. The state s contains POVM probabilities of the current and target reduced states plus the scalar bath parameters. But the QSD dynamics in Eqs. (2)-(3) carry memory through O_z and O_w, which are not in s. The loop in Fig. 1 re-encodes only rho_ini = rho_n, so both agents are solving a memoryless POMDP. The paper presents the crossover as a general fact about DRL versus SL, but it may be an artifact of this restricted observation space. In the low-gamma regime, which the paper highlights as strongly non-Markovian, the memory is largest and the conclusion is least secure.\n\nThird, minor but annoying: Appendix A has apparent typos. Eqs. (A.8)-(A.9) use mismatched operators in the commutators. The final master equation is cited, so the numerics may be unaffected, but the derivation should be fixed. Also, Fig. 3 shows no error bars, the value of h in Eq. (4) is never given, and the SL local-optimality filter is described only qualitatively.\n\nWho this is for: people doing ML-based quantum control in simulations. It is not a hardware demo and the impact is subfield-level.\n\nMy recommendation: send it to peer review, but require major revision. With the reward defined, error bars added, and the memory issue at least discussed — or the state extended to include O_z and O_w — the crossover claim could be solid. As written, the missing details prevent reproduction and the observation-space limitation undermines the generality.","headline":"A plausible crossover between DRL and SL for noise-aware pulse design, but the DRL reward is unspecified and the policy state drops non-Markovian bath memory, so the claim needs revision before it can be trusted.","tokens_in":17279,"tokens_out":4377,"would_cite":false,"duration_ms":39971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes noise-aware machine-learning pulse design for arbitrary qubit state preparation, reporting that a single model with environmental parameters as input features outperforms noise-blind control and generalizes to unseen…","keywords":["quantum state preparation","quantum control","deep reinforcement learning","supervised learning","non-Markovian environment","noise suppression","open quantum systems","pulse design"],"falsifier":"Train a Case-3 DRL model with bath frequencies $\\gamma$ at or above a moderate value, then test it at a strongly non-Markovian value such as $\\gamma = 0.1$, and compare its average fidelity with a per-environment Case-2 model trained at $\\gamma = 0.1$ using the same QSD simulator; if the Case-3 model falls clearly below the Case-2 model, the sufficiency of $s$ and the generalization claim fail in the non-Markovian regime.","tokens_in":16253,"feed_emoji":"⚛️","tokens_out":8336,"duration_ms":69113,"temperature":0.7,"pith_summary":"The paper asks whether a quantum control policy can be made noise-aware by feeding the environment's parameters directly into the neural network that chooses control pulses. Using arbitrary qubit-to-qubit state preparation in a two-level system coupled to a bosonic bath, it trains Deep Reinforcement Learning (DRL) and Supervised Learning (SL) models under three conditions: ignoring noise, incorporating noise parameters only in the simulation, and encoding noise parameters as input features. It reports that noise-aware models achieve higher test-set fidelity than noise-unaware models, and that a single feature-encoded model performs about as well as separately retrained per-environment models, including on environment parameters not seen in training. It also reports a regime split: DRL outperforms SL in weak or absent noise, while SL is more stable and outperforms DRL in strong noise. If correct, this gives a practical rule for choosing the learning method by noise level and for replacing many environment-specific controllers with one general network.","feed_headline":"Noise-aware models beat noise-blind pulse design for qubits","feed_subtitle":"Training on bath parameters lifts qubit fidelity; use DRL for weak noise, supervised learning for strong noise.","key_machinery":"The load-bearing object is the state vector $s = [P_{\\mathrm{ini}}^1,\\ldots,P_{\\mathrm{ini}}^4,P_{\\mathrm{tar}}^1,\\ldots,P_{\\mathrm{tar}}^4,\\Gamma,\\gamma,T]$, built by compressing the current and target density matrices into four POVM probabilities each and appending the bath parameters. This vector is the input on which both the DRL policy and the SL classifier act; at each step the simulator evolves the density matrix with the non-Markovian quantum-state-diffusion master equation under a Lorentz-Drude spectral density. The claim that Case 3 works rests on this vector carrying enough information to choose the next pulse.","core_discovery":"The central claim is that incorporating the bath parameters $\\Gamma$, $\\gamma$, and $T$ into the training loop—either as fixed inputs to the simulation (Case 2) or as explicit input features to the network (Case 3)—yields higher average state-preparation fidelity than a model trained without any noise information (Case 1). For the same neural network architecture, the DRL models outperform SL models in noise-free or weakly noisy settings, while SL models retain higher fidelity when noise is strong; the crossing point is the paper's main comparative finding. The paper further claims that a Case-3 model trained over a range of environmental parameters can design pulses for unseen parameter values, so that one network acts as a universal controller across the covered noise regime.","pith_inferences":["The near-equal performance of Case 2 and Case 3 suggests the practical advantage of encoding environment parameters as features is logistical (one model for many environments) rather than a fidelity gain; an engineering rule would be to use Case 3 for deployment across many environments and Case 2 when inference speed matters.","The sufficiency of $s$ is the most exposed assumption: adding bath-memory observables to the state, or using a recurrent policy, is a direct test of whether the method extends deeper into the non-Markovian regime.","The same training recipe could be applied to other spectral densities, such as Ohmic or sub-Ohmic baths; the QSD equations would change but the input-encoding idea would not.","Because the action set and step duration are discrete, the policies are approximate optimal controllers; allowing continuous pulse amplitudes or adaptive time steps could reveal how much fidelity is lost to discretization."],"forward_implications":["A single Case-3 network replaces the need to retrain a policy for each new set of environmental parameters within the covered range.","Noise-aware training pays off most when noise is strong, because the fidelity gap between Case-1 and Cases-2/3 grows with $\\Gamma$, $\\gamma$, and $T$.","In low-noise or noise-free devices, DRL-trained pulses should be preferred; in high-noise devices, SL-trained pulses should be preferred.","SL models are more stable and faster at inference time, with smaller average step counts, while DRL models are more time-consuming and show more scattered design times.","The comparison is made with identical network architectures, so the reported differences are attributed to the learning algorithms and the input encoding rather than to model capacity."],"supporting_citations":[{"why":"Supplies the non-Markovian master equation and the closed equations for the noise operators that the simulator solves.","marker":"[53]"},{"why":"Provides the earlier adaptive-denoising state-preparation strategy that parameterizes the environment and the SL training and data-filtering approach extended here.","marker":"[46]"},{"why":"Foundational quantum-state-diffusion method used to derive the system-bath evolution.","marker":"[52]"},{"why":"Supplies the DRL dynamic-pulse-control approach for universal quantum state preparation that the DRL implementation builds on.","marker":"[9]"},{"why":"Provides the Deep Q-Network algorithm, including the target network, experience replay, and loss used in training.","marker":"[66]"},{"why":"Introduces the experience-replay mechanism that stabilizes the DRL training updates.","marker":"[67]"},{"why":"Supports the fidelity-evaluation filtering used to remove locally optimal actions from the SL training data.","marker":"[76]"},{"why":"Supplies the Pauli-4 POVM operators used to encode density matrices as probability vectors in the state s.","marker":"[85]"}],"fun_headline_variants":["AI designs noise-aware pulses for qubit state prep","DRL wins at low noise, supervised learning at high","Neural nets tune pulses using bath parameters for fidelity","Machine learning tailors qubit pulses to environmental noise","Pulse design learns from noise: DRL vs supervised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The network's input contains only the current-state and target-state measurement probabilities plus the three bath parameters, with no information about the bath's memory, so the whole scheme assumes this short vector is enough to choose the optimal next pulse even in the non-Markovian regime.","fun_headline_variants_meta":{"raw":{"variants":["AI designs noise-aware pulses for qubit state prep","DRL wins at low noise, supervised learning at high","Neural nets tune pulses using bath parameters for fidelity","Machine learning tailors qubit pulses to environmental noise","Pulse design learns from noise: DRL vs supervised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1443,"prompt_tokens":873,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":492}},"tokens_in":489,"tokens_out":570,"duration_ms":5554,"temperature":1.0,"reasoning_tokens":492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:46:20.124035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a Case-3 DRL model with bath frequencies $\\gamma$ at or above a moderate value, then test it at a strongly non-Markovian value such as $\\gamma = 0.1$, and compare its average fidelity with a per-environment Case-2 model trained at $\\gamma = 0.1$ using the same QSD simulator; if the Case-3 model falls clearly below the Case-2 model, the sufficiency of $s$ and the generalization claim fail in the non-Markovian regime.","supporting_citations":[{"cited_title":"Quantum state transmission through a spin chain in finite- temperature heat baths","cited_arxiv_id":null,"evidence_quote":"Supplies the non-Markovian master equation and the closed equations for the noise operators that the simulator solves."},{"cited_title":"Adaptive denoising quantum state preparation in a dynamic environment","cited_arxiv_id":null,"evidence_quote":"Provides the earlier adaptive-denoising state-preparation strategy that parameterizes the environment and the SL training and data-filtering approach extended here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundational quantum-state-diffusion method used to derive the system-bath evolution."},{"cited_title":"Deep reinforcement learning for universal quan- tum state preparation via dynamic pulse control","cited_arxiv_id":null,"evidence_quote":"Supplies the DRL dynamic-pulse-control approach for universal quantum state preparation that the DRL implementation builds on."},{"cited_title":"Human-level control through deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the Deep Q-Network algorithm, including the target network, experience replay, and loss used in training."},{"cited_title":"Enhanced quan- tum state preparation via stochastic predictions of neural networks","cited_arxiv_id":null,"evidence_quote":"Supports the fidelity-evaluation filtering used to remove locally optimal actions from the SL training data."},{"cited_title":"Reconstructing quantum states with generative models","cited_arxiv_id":null,"evidence_quote":"Supplies the Pauli-4 POVM operators used to encode density matrices as probability vectors in the state s."}],"review_version":2}