{"id":"7f5669bf-85a8-4c93-a239-a52a97041300","arxiv_id":"2509.04265","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Reinforced SDMD uses RL to select trajectory initial conditions guided by a spectral-consistency reward, with qualitative experiments on three stochastic test systems and standard bandit/DQN/PPO machinery.","lead":"This paper proposes letting a reinforcement learning agent decide where to start trajectories for Koopman spectral analysis of stochastic systems, rewarding regions where the estimated eigenpairs look self-consistent. The authors test the scheme on double-well, Duffing, and FitzHugh-Nagumo systems and add an error-propagation analysis for the bandit, DQN, and PPO algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The spectral-consistency reward (Eq. 3.4) is never shown to correlate with true Koopman estimation error; maximizing it may simply avoid regions where the current SDMD model is wrong, so the central claim lacks support.","rationale":"The reader's weakest assumption—that the reward correlates with true spectral error—is exactly the load-bearing point. I agree. The paper's experiments only show that the learned reward map highlights potential minima and that eigenfunctions look qualitatively plausible; there is no quantitative comparison against uniform/random sampling, no use of the analytical FHN eigenvalues, and no seed averaging. The theory section is conditional: Assumption 5.2 already assumes the reward estimates are accurate enough to distinguish the best arm, which is essentially the conclusion the central claim needs. Since the reward is a self-consistency residual of the current model, the direction of the bias is known: it favors regions where the model currently performs well and avoids regions where it fails. The only counterweight is the exploration bonus, but its strength is arbitrary and no sensitivity analysis is given. A missing code link (Appendix A.1) further prevents independent verification. These are not mere presentation issues; they leave the central claim without supporting evidence. The proposed test—comparing RL-selected data against uniform initialization on a system with known/reference spectra—would settle whether the reward actually improves estimation. If it does not, the paper's main conclusion should be rejected or substantially weakened. Hence the reader's REJECT verdict stands.","tokens_in":16445,"tokens_out":4992,"duration_ms":46684,"concrete_test":"On the FitzHugh–Nagumo system of Sec. 4.3, obtain reference eigenpairs (e.g., from the quoted analytical eigenvalues λ_n = −n^2 μ + i n ω for the noiseless system, or from a long unbiased simulation with the same noise). Run Bandit-SDMD (Algorithm 1) with reward Eqs. (3.4)–(3.5) and the paper's hyperparameters; then measure the error of the leading 2–5 estimated eigenvalues/eigenfunctions against the reference. Run the same SDMD code with uniform random trajectory initialization using the same total data budget, dictionary, and number of trajectories. Repeat over at least 10 seeds and report mean/standard deviation. If RL-selected data does not reduce the eigenvalue/eigenfunction error below the uniform-random baseline (e.g., by a nontrivial margin), the spectral-consistency reward is not selecting regions that improve the true estimate, and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the reward in Eqs. (3.4)–(3.5), computed from the current SDMD estimate, guides sampling toward regions that reduce the true error of the leading Koopman eigenpairs. No argument or experiment in the paper establishes this link. Eq. (3.4) measures the residual of the estimated eigenpair equations on newly generated data. Since the estimated eigenpairs were fitted to previously collected data, a low residual on new data from a region only says the model extrapolates consistently there; it does not say the eigenpairs are accurate. Regions where the model is inaccurate yield high residuals, hence low reward, so the agent is rewarded for avoiding exactly the regions that would supply corrective data. The exploration bonus in Eq. (3.5) is the only push toward underexplored regions, but α_exp is hand-chosen and no analysis shows it overcomes this avoidance. The theory does not repair the gap: Assumption 5.2 postulates |R̂_a − R_a| ≤ ε_sdmd for a 'true reward' R_a that is never defined, and since the reward is recomputed as the SDMD model evolves, the per-arm rewards are non-stationary, so Theorem 5.3's stationary-bandit regret bound does not apply. Theorems 5.6 and 5.8 likewise only propagate an assumed SDMD error bound through standard AVI/API arguments; they do not show that the learned sampling policy improves the eigenpair estimate. Thus the central claim—that RL-selected initialization improves estimation of leading Koopman eigenpairs—is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reinforced SDMD, a method that couples Stochastic Dynamic Mode Decomposition with reinforcement learning to choose trajectory initialization regions. An RL agent (multi-armed bandit, DQN, or PPO) selects spatial regions, and a reward combining spectral consistency of the current SDMD eigenpairs with an exploration bonus guides the data acquisition. The authors present qualitative experiments on a double-well potential, a stochastic Duffing oscillator, and the FitzHugh--Nagumo model, and provide a convergence analysis that propagates a bounded SDMD operator error through bandit, approximate value iteration, and approximate policy iteration bounds. The central claim is that the learned sampling policy improves estimation of the leading Koopman eigenpairs.","tokens_in":16813,"tokens_out":4932,"duration_ms":46379,"significance":"If the central claim were established, the paper would address an important practical problem: where to collect trajectory data for Koopman spectral estimation. The integration of RL with SDMD is original, and the pseudocode and theoretical error-propagation framework are useful starting points. The paper's theory is internally consistent under its assumptions, but the assumptions and the stated reward do not connect to the central claim. The experimental sections provide only qualitative evidence and no comparison with standard sampling strategies. The significance of the contribution therefore depends entirely on whether the reward actually improves true spectral estimation error, and this is not demonstrated.","major_comments":[{"comment":"The central claim that RL-guided initialization improves estimation of leading Koopman eigenpairs is not tested quantitatively. There is no comparison against random or uniform trajectory initialization, no error metric for eigenvalues or eigenfunctions, and no error bars or multiple-seed analysis. For the FHN experiment, the analytic eigenvalues λ_n = -n^2 μ + i n ω are mentioned in §4.3 but are never compared with the numerical eigenvalues in Fig. 6; moreover those formulas are stated for the noiseless system, so their applicability to the stochastic experiment is unclear. The figures are qualitative snapshots and do not support the causal statement that the learned policy improves spectral estimates.","section":"Section 4, Figures 2, 3, 5, 6"},{"comment":"The spectral-consistency reward is computed with the current SDMD estimate. A low value of L_Spectral_Consistency on newly generated data only shows that the new region is consistent with the current estimate; it does not show that the estimated eigenpairs are accurate. Regions where the current model is inaccurate would produce high residuals and hence low reward, so maximizing this reward may steer sampling away from the very regions that would supply corrective data. The exploration bonus is a hand-tuned counterweight, and no analysis or experiment demonstrates that it overcomes this avoidance. Without this link, the central claim is unsupported.","section":"Section 3.2, Eqs. (3.4)-(3.5)"},{"comment":"The regret analysis is disconnected from the implemented algorithm. The true reward R_a is never defined in terms of Koopman spectral estimation error; even if a stationary reward existed, Assumption 5.2 assumes ε_sdmd < Δ_min/2, which is effectively the conclusion needed to guarantee that the reward estimates resolve the best arm. In the actual algorithm, per-arm rewards are recomputed as the SDMD model evolves and include a time-varying exploration bonus, so the arms are non-stationary and the stationary-bandit regret bound does not apply to the implemented procedure.","section":"Section 5.1, Assumption 5.2 and Theorem 5.3"},{"comment":"These theorems propagate a generic SDMD operator-error bound ε_sdmd through standard approximate value iteration and approximate policy iteration arguments. They do not show that the RL-selected sampling reduces ε_sdmd, nor that the learned policy improves the leading Koopman eigenpairs. Consequently, the theoretical analysis does not establish the paper's stated claim that the final estimation accuracy is linked to the quality of the learned sampling policy.","section":"Sections 5.2-5.3, Theorems 5.6 and 5.8"}],"minor_comments":[{"comment":"The expression L_Spectral_Consistency appears to be a formatting or notation error; define a clear symbol such as L_sc and specify the norm used explicitly.","section":"Eq. (3.4)"},{"comment":"The reward map is only visually compared with the potential landscape; a quantitative correlation measure would strengthen the claim that the algorithm rediscovers the potential well structure.","section":"Section 4.1"},{"comment":"The source code link is missing; either include the link or remove the reproducibility statement.","section":"Appendix A.1"},{"comment":"The analytic eigenvalues are stated for the noiseless FHN system while the experiment includes noise; explain how these formulas are used as ground truth or clarify their role.","section":"Section 4.3"},{"comment":"The state is a sliding window of starting points, but the reward is computed after running SDMD on an accumulated dataset; the pseudocode does not specify how often SDMD is retrained or how the dictionary is updated over time.","section":"Algorithms 2 and 3"}],"recommendation":"reject","confidential_remarks":"The central loop of the method—using the current SDMD model's self-consistency as the learning signal—is not shown to correlate with true Koopman estimation error, and the theory assumes away precisely the property it needs to prove. I do not see how a revision within the manuscript's scope can repair this without a fundamentally different reward definition or a very different set of experiments. If the editors prefer a softer outcome, a major-revision invitation could be considered only if the authors reframe the contribution as a heuristic and provide quantitative validation against baselines, but as it stands the load-bearing gap is too large."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zeb — quick take on Xu et al. (arXiv:2509.04265). The piece is a plausible first cut at a real problem: Koopman spectral estimates depend on where you collect trajectories, and nobody has a principled way to choose starting points. The authors wire SDMD to a bandit/DQN/PPO agent that picks initial regions, with a spectral-consistency reward plus an exploration bonus. That's new, and the experiments on double-well, Duffing, and FitzHugh–Nagumo show the agent visibly concentrates samples near attractors and basins. The paper is clearly written, and the theory is honest — Assumption 5.2 is stated outright, and the failure case is discussed.\n\nBut the central claim — that RL-guided sampling improves the eigenpair estimates — is not actually tested. There is no comparison against uniform or random sampling, no use of the FHN analytical eigenvalues as ground truth, and no error bars over seeds. The plots are qualitative progressions of eigenfunctions. The reward itself is the biggest conceptual issue: Eq. (3.4) scores new data by how well the current SDMD model's eigenpairs reproduce it. If the model is wrong in some region, that region scores poorly and the agent learns to avoid exactly the data that would correct it. The exploration bonus is the only counterweight, and its strength is a hand-picked constant. The theory doesn't close the gap: Theorem 5.3 just restates the bandit result under the assumption that the SDMD reward error is smaller than half the gap — which is close to assuming the conclusion — and the bandit setting assumes stationary per-arm rewards while the actual rewards are recomputed with an evolving model. The code link is a placeholder.\n\nNone of this kills the underlying idea. The combination is new and the experiments are suggestive. But as it stands, the paper's headline claim is unsupported. It needs a simple random-sampling baseline, a quantitative error comparison against known spectra, seed averaging, and a serious discussion (or experiment) on whether spectral consistency correlates with true estimation error.\n\nWho's this for? People working on data acquisition for Koopman methods or active learning for dynamical systems. I'd send it to review — the idea deserves referee time — but I'd expect the referees to demand the missing baseline before acceptance.","headline":"RL-guided sampling for Koopman estimation is a new idea, but the paper doesn't show it actually helps: no baseline, no ground truth, and a self-consistency reward that may just keep the model comfortable.","tokens_in":17373,"tokens_out":2602,"would_cite":false,"duration_ms":24320,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Reinforced SDMD, a data-acquisition method in which a reinforcement-learning agent chooses where trajectories start so that Stochastic Dynamic Mode Decomposition better estimates the leading Koopman eigenvalues and eigen","keywords":["Koopman operator","stochastic dynamical systems","stochastic dynamic mode decomposition","reinforcement learning","spectral consistency","spectral estimation","metastability","data acquisition"],"falsifier":"Run Reinforced SDMD on a double-well system starting from a deliberately poor SDMD model that is wrong near one well. If the learned policy keeps sampling the well the model already fits because spectral-consistency reward is high there, while final eigenvalue error stays large or worsens compared with uniform sampling, then the reward is not tracking true estimation error and the central claim fails.","tokens_in":16219,"feed_emoji":"🎯","tokens_out":8693,"duration_ms":81484,"temperature":0.7,"pith_summary":"Koopman spectral analysis turns a nonlinear stochastic system into a linear evolution of observables, but the accuracy of standard estimators depends on where trajectory data happen to be collected. This paper proposes to stop treating data collection as a fixed input: a reinforcement-learning agent chooses the starting points of new trajectories, Stochastic Dynamic Mode Decomposition updates an approximation of the Koopman semigroup, and a spectral-consistency reward tells the agent how well the current eigenpairs predict evolution on the newly gathered data. On double-well, Duffing, and FitzHugh–Nagumo systems, the learned policies concentrate sampling near potential minima, basins of attraction, and slow manifolds without prior dynamical knowledge, and the leading eigenfunctions settle into the expected metastable structure. The paper also proves convergence bounds linking the final policy quality to the SDMD operator error. If correct, the framework turns passive spectral analysis into an active search: with the same data budget, practitioners can get better estimates of decay rates, oscillations, and metastable behavior.","feed_headline":"Reinforcement learning steers sampling to better Koopman spectra","feed_subtitle":"An agent rewarded for spectral consistency spends the data budget on dynamically important regions, improving eigenpair estimates.","key_machinery":"The load-bearing mechanism is the spectral-consistency reward R_t = R_0 − L_SC + α_exp / (η(x_new) + ε), where L_SC = Σ_i ||φ_i(Y) − μ_i φ_i(X)||²_ρ measures how well the SDMD-estimated eigenpairs (μ_i, φ_i) reproduce the observed evolution of new trajectory data, and η is a kernel density estimate over past starting points. This reward links the RL policy to Koopman spectral quality: it tells the agent which regions make the current eigenpairs predictive. It is balanced by the exploration bonus α_exp/η so the agent does not collapse onto a single well-sampled region. SDMD supplies the operator estimate and hence the eigenpairs entering the reward.","core_discovery":"The central claim is that the quality of a data-driven Koopman spectral estimate is not fixed by the dictionary or the estimator alone; it can be improved by choosing where the trajectories start, and this choice can be learned. Reinforced SDMD formulates that choice as a reinforcement-learning problem: the action space is a partition of the state space into boxes, the state is a sliding window of recent starting points, and the reward is the spectral-consistency error of the current SDMD eigenpairs on the new data plus an exploration bonus. The paper reports that the learned Q-values and policies rediscover the double-well potential landscape, the two basins of the Duffing oscillator, and t","pith_inferences":["A testable extension not run in the paper is to compare Reinforced SDMD against an active-learning baseline that samples where the current model's predictions are most uncertain; disagreement between the two criteria would reveal whether spectral consistency is a proxy for information gain or a separate objective.","The convergence analysis treats the per-region reward as stationary, but in the implemented algorithm the SDMD model is updated as data accumulate, so the reward landscape drifts; non-stationary bandit analysis or freezing the model during acquisition would tighten the match between theory and practice.","The learned reward/Q-map could serve as a cheap pre-sampling diagnostic: a short Reinforced SDMD run identifies dynamically important regions before a full Koopman analysis, and this information may transfer to similar systems.","Because the action space is a grid of boxes, cost grows with dimension; a continuous-action variant, identified by the authors as future work, would be the natural next step toward higher-dimensional systems."],"forward_implications":["Data collection for Koopman spectral estimation can be automated: an RL agent with no prior dynamical knowledge learns which regions carry spectral information and biases trajectory initializations there.","Given a fixed data budget, concentrating trajectories in reward-identified regions should reduce error in the leading Koopman eigenvalues and eigenfunctions compared with spatially uniform sampling.","The final estimation accuracy is bounded by the SDMD error ε_sdmd: once the sampling policy is good, the remaining bottleneck is the operator estimator itself, motivating better estimators alongside better sampling.","The same coupling works for stateless bandit, deep Q-learning, and policy-gradient RL, so the framework is not tied to one learning algorithm.","The theoretical bounds mean the sampling policy's quality directly enters the spectral estimation guarantee, justifying the RL-loop design."],"supporting_citations":[{"why":"Supplies the SDMD estimator and its error bound ε_sdmd; its eigenpairs enter the reward and its error controls all convergence theorems.","marker":"[41]"},{"why":"Provides the EDMD dictionary-based operator approximation that SDMD extends to stochastic dynamics.","marker":"[39]"},{"why":"Defines the multi-armed bandit and ε-greedy update used in Bandit-SDMD.","marker":"[33]"},{"why":"Introduces DQN, experience replay, and target networks used in DQN-SDMD.","marker":"[25]"},{"why":"Introduces PPO and the clipped surrogate objective used in PPO-SDMD.","marker":"[32]"},{"why":"Supplies the approximate value iteration error bound used to prove DQN-SDMD convergence.","marker":"[26]"},{"why":"Supplies the simulation lemma connecting Q-function error to policy suboptimality in the DQN proof.","marker":"[17]"}],"fun_headline_variants":["RL guides sampling for more accurate Koopman spectra","Reinforcement learning picks where to sample for better Koopman spectra","RL chooses start regions to improve Koopman eigenpair estimates","Smart data acquisition via RL improves Koopman spectral estimates","Reinforcement learning directs sampling for better Koopman spectra"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole method depends on the assumption that rewarding the current model for predicting new data well will push the sampler toward regions that make the model more accurate—not merely toward regions where the model is already comfortable.","fun_headline_variants_meta":{"raw":{"variants":["RL guides sampling for more accurate Koopman spectra","Reinforcement learning picks where to sample for better Koopman spectra","RL chooses start regions to improve Koopman eigenpair estimates","Smart data acquisition via RL improves Koopman spectral estimates","Reinforcement learning directs sampling for better Koopman spectra"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2340,"prompt_tokens":758,"completion_tokens":1582,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1497}},"tokens_in":502,"tokens_out":1582,"duration_ms":11981,"temperature":1.0,"reasoning_tokens":1497,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:15:16.707875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Reinforced SDMD on a double-well system starting from a deliberately poor SDMD model that is wrong near one well. If the learned policy keeps sampling the well the model already fits because spectral-consistency reward is high there, while final eigenvalue error stays large or worsens compared with uniform sampling, then the reward is not tracking true estimation error and the central claim fails.","supporting_citations":[{"cited_title":"A Data-Driven Framework for Koopman Semigroup Estimation in Stochastic Dynamical Systems","cited_arxiv_id":"2501.13301","evidence_quote":"Supplies the SDMD estimator and its error bound ε_sdmd; its eigenpairs enter the reward and its error controls all convergence theorems."},{"cited_title":"Introduction to multi-armed bandits.Foundations and Trends® in Machine Learning, 12 (1-2):1–286, 2019","cited_arxiv_id":null,"evidence_quote":"Defines the multi-armed bandit and ε-greedy update used in Bandit-SDMD."},{"cited_title":"Near-optimal reinforcement learning in polynomial time.Machine learning, 49(2):209–232, 2002","cited_arxiv_id":null,"evidence_quote":"Supplies the simulation lemma connecting Q-function error to policy suboptimality in the DQN proof."}],"review_version":1}