{"id":"c66813e6-7198-40e4-85de-1fbc119e28c8","arxiv_id":"2509.08241","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Recursive least-squares updates of Koopman models let a model-predictive controller learn a soft-robot balancing task with about one-eightieth of the training time used by a reinforcement-learning baseline.","lead":"This paper introduces a control-learning pipeline that updates Koopman linear models in real time with recursive least squares, and tests it on a simulated robot arm and a soft Stewart platform. It reports matching or beating reinforcement learning baselines with a fraction of the data and training time, and gives a convergence proof for the model update under Markov-chain assumptions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The convergence proof assumes online data form a fixed ergodic Markov chain, but RKL's MPC policy changes with every model update, so the closed-loop transition kernel is time-varying and the SLLN in Appx D does not apply to the online loop.","rationale":"The reader's weakest_assumption identifies the same gap I consider most load-bearing: the formal convergence proof applies to a time-homogeneous ergodic Markov chain, while RKL's online closed-loop process is nonstationary because the model and therefore the MPC policy change at every step. This is not a minor technicality; it means the headline theoretical contribution—convergence of the online model estimate—is not actually proven for the algorithm as run. The standalone batch-EDMD consistency result in Appx D is mathematically sound, and the exact RLS/EDMD equivalence in Appx B is correct under the stated full-rank conditions. The empirical results and open-source code are credible evidence that the method works in practice, which is why I do not recommend rejection. However, the paper overclaims by saying it provides convergence guarantees for RKL. The correct fix is to state the theorem as a batch consistency result, add conditions under which the online process becomes asymptotically stationary (e.g., policy convergence plus a mixing condition), or provide a separate nonstationary convergence analysis. The reader's CONDITIONAL verdict already captures this, so I would leave the verdict unchanged.","tokens_in":20785,"tokens_out":8679,"duration_ms":119464,"concrete_test":"Run RKL-SAC exactly as in Algorithm 1 on the planar two-link arm for N=10^5 online steps with a fixed random seed. Every 10^4 steps, compute the batch EDMD/RLS estimate K_N from all data so far, and separately generate a 10^6-step trajectory with the policy frozen at that point to estimate the stationary target K*_final = E[phi(f_final(s)) phi(s)^T] (E[phi(s) phi(s)^T])^{-1}. Track the normalized Frobenius error ||K_N - K*_final|| / ||K*_final||. If this error does not decay to a small value (e.g., <5%) as N grows, the online RKL process is not converging to the stationary limit required by the Appx D proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing theoretical claim is that the EDMD/RLS estimate K_N converges to K* (Appx D.3, Eq. 39). The proof invokes the SLLN for Markov chains under sufficient conditions that include {s_k} being an irreducible, aperiodic, positive recurrent Markov chain with a single invariant measure mu. In Algorithm 1, s_k=(x_k,u_k) is generated in closed loop: u_{k+1}=MPC(K_{k+1}, z_{k+1}, zbar), and K_{k+1} is updated at every step via Eq. 7. Therefore the transition kernel of the data process depends on K_k, which changes with each sample. The process is not time-homogeneous and has no fixed invariant measure mu to which the empirical averages in Eqs. 30-31 converge. The only justification, stated in Appx D.2, is that trajectories 'can be approximated as Markov chains'—an assertion, not a proven condition, and insufficient for a formal SLLN. Moreover, the initial dataset is collected with random actions or a PD controller (Sec. E.1.3) while online data use MPC, so even the concatenated dataset is generated by more than one transition kernel. What Appx D proves is batch-EDMD consistency for a fixed autonomous/stochastic system, not convergence of the adaptive RKL loop. The empirical results may remain valid, but the formal convergence guarantee for the online algorithm is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Recursive Koopman Learning (RKL), an online control-policy learning pipeline that combines an EDMD-initialized Koopman model with RLS-based recursive model updates and MPC. The central claims are: (i) one RLS update is algebraically equivalent to retraining the full EDMD model on the appended dataset; (ii) under ergodic Markov-chain conditions, the EDMD/RLS estimate converges almost surely to the optimal finite-dimensional Koopman estimate; (iii) the per-step update cost is O(n^2) and independent of dataset size; and (iv) on a simulated planar two-link arm and a soft Stewart platform, RKL achieves substantially higher sample efficiency than RL and NN-MPPI baselines. The paper also introduces a 'weak ACG hypothesis' to explain why data collected while attempting the control task are informative for model learning.","tokens_in":21158,"tokens_out":7901,"duration_ms":87911,"significance":"The standalone batch-EDMD consistency argument in Appendix D is a correct application of the Markov-chain SLLN under the stated assumptions, and the RLS equivalence proof in Appendix B is elegant. The complexity analysis is straightforward and the open-source C++ implementation, together with the hardware experiments, is a useful empirical contribution. If the online convergence guarantee were valid, this would be a significant result for sample-efficient Koopman-based control. However, the main theoretical claim—convergence of the online RKL loop—is not established by the supplied analysis because the closed-loop data are not a time-homogeneous Markov chain; the paper proves a batch-consistency result for a fixed system, not convergence of the adaptive online algorithm.","major_comments":[{"comment":"The convergence proof applies the Markov-chain SLLN (Eq. (34)) to the data sequence {s_k}. This requires a fixed irreducible, aperiodic, positive-recurrent chain with a single invariant measure μ. In Algorithm 1, s_k=(x_k,u_k) and u_{k+1}=MPC(K_{k+1}, z_{k+1}, \\bar{z}), with K_{k+1} updated at every step by Eq. (7). Therefore the transition kernel of {s_k} is time-varying and there is no fixed invariant measure to which the empirical averages in Eqs. (30)–(31) converge. The statement in D.2 that trajectories 'can be approximated as Markov chains' is an assertion, not a derived sufficient condition. Moreover, the initial dataset is generated by a PD controller or random actions (Sec. E.1.3), while online data use the MPC policy, so the concatenated dataset is generated by more than one transition kernel. The argument establishes batch-EDMD consistency for a fixed system, not convergence o","section":"Appendix D.2–D.3, Eqs. (30)–(39) and Algorithm 1"},{"comment":"The assumptions of irreducibility, aperiodicity, and positive recurrence are not justified by the experiments. The text argues these follow from controllability of the open-loop system, but the chain is generated by the closed-loop MPC policy, which is typically stabilizing and may have an attractor. A deterministic closed-loop map is not an irreducible Markov chain on the full state-action space unless stochastic exploration is explicitly injected. If exploration noise is present, its distribution and variance must be specified in the model; if not, the SLLN hypotheses fail even for a fixed policy.","section":"Appendix D.2"}],"minor_comments":[{"comment":"The notation \\bar{z} is used inconsistently: in Algorithm 1 it denotes the control goal, while in Algorithm 3 and the surrounding text it appears to denote the next observation z_{k+1}. This makes the RLS update description confusing and should be clarified.","section":"Appendix B and Algorithm 3"},{"comment":"The recursive update of P assumes P_k is invertible, not merely a pseudoinverse. The paper states full rank as an assumption, but this should be stated clearly before Eq. (6), and the exact-arithmetic caveat for the claimed 'same result as EDMD' should be mentioned.","section":"Section 3.3 and Eq. (6)"},{"comment":"Results are reported as means over 50 trials without variance or confidence intervals. Several adjacent rows in Table 1 have very close RMSE values (e.g., RKL-SAC 3,000+500 vs 5,000+500), so statistical significance of the ranking is unclear.","section":"Section 4.2, Table 1"},{"comment":"The convergence result is presented as an informal derivation rather than a formal theorem. Stating it as a theorem with explicitly numbered hypotheses and a proof would improve verifiability and make the gap between the batch result and the online loop more visible.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are strong and the open-source code is valuable, but the theoretical contribution as written overclaims: the convergence proof is for batch EDMD on a fixed Markov chain, not for the adaptive online RKL closed loop. I would ask the authors to either prove a convergence result for time-varying/stochastic-approximation-type updates or to explicitly reframe the theoretical contribution as a batch-consistency result and state that online convergence remains an open problem. The 'weak ACG hypothesis' also needs a precise definition if it is to support the theoretical narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper with a real empirical claim and a modest theoretical contribution. The RLS-EDMD equivalence is not new — the paper cites Zhang et al. and Calderon et al. — but the formal consistency result for EDMD/RLS estimators under an ergodic Markov chain (Appx D) is a clean, correct application of Breiman's SLLN, and it is stated with the right conditions. The complexity analysis showing O(n^2) per update is correct. The open-source C++ code and hardware validation on the soft Stewart platform are real assets; the sample-efficiency numbers (1 min 20 sec vs 2h 46m for SAC) are striking if they hold up.\n\nNow the soft spots. The stress-test is on point. The convergence proof assumes the data process {s_k} is a time-homogeneous, ergodic Markov chain with fixed invariant measure. In Algorithm 1, the MPC policy is recomputed after every RLS update, so the closed-loop transition kernel changes with k. The concatenated dataset also mixes random/PD-collected initial data with MPC-driven online data. Appx D.2 simply asserts trajectories 'can be approximated as Markov chains' and that experimental results indicate feasibility. That is not enough. What Appx D actually proves is batch-EDMD consistency for a fixed system; it does not establish convergence of the adaptive RKL loop. The paper does hedge in the main text by calling the conditions 'sufficient' and linking them to the ACG hypothesis, but the headline claim is stronger than what is proven.\n\nThe experiments are also a bit selective: the N-tracking results are best-of-five trials, no error bars; the RL baseline on hardware doesn't receive the demonstration dataset, so it is not an apples-to-apples comparison. The simulation results are more thorough (50 trials, multiple seeds for baselines). The limitations section is honest about the imbalance issue when the puck stays still, which is consistent with the ergodicity gap.\n\nWho is this for? People building online Koopman controllers on real robots. They will find the RLS update and the complexity argument useful. The theory is not the reason to read the paper; the recipe and the hardware demonstration are. It deserves a serious referee — the issues are addressable (verify or weaken the online claim, add error bars, fair the RL baseline) but they need to be fixed before acceptance.","headline":"The RLS-EDMD equivalence is old but the batch consistency theorem is new; the stress-test is right that the online loop is not covered by the Markov-chain SLLN, but the empirical pipeline is still worth a look.","tokens_in":21632,"tokens_out":1948,"would_cite":true,"duration_ms":23557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recursive Koopman Learning: one recursive least squares update equals full EDMD retraining, enabling real-time sample-efficient control.","keywords":["Koopman operator","recursive least squares","online model update","extended dynamic mode decomposition","Markov chain convergence","sample-efficient control","model predictive control","soft robotics"],"falsifier":"Take a fixed recorded data stream, compute the batch EDMD Koopman matrix once, then feed the same snapshots one by one through the RLS updates and compare the final matrices; the paper's identity predicts equality up to floating-point roundoff. A concrete way to observe failure: if the inverse covariance P becomes ill-conditioned after many similar states, recursive and batch estimates will diverge. Separately, during an RKL run on hardware, record the empirical distribution of (state, action) pairs over time; if that distribution keeps shifting as the policy updates rather than settling down,","tokens_in":20703,"feed_emoji":"🔄","tokens_out":7917,"duration_ms":87018,"temperature":0.7,"pith_summary":"Recursive Koopman Learning (RKL) claims that an online controller can update its learned dynamics model at every time step—at negligible computational cost—without ever losing the quality of a full batch refit. The central analytical result is that one recursive least squares update produces exactly the same Koopman matrix as retraining the entire extended dynamic mode decomposition model on all data collected so far, so the online model is always the globally optimal least-squares fit. Under specified ergodicity conditions on the data stream, the paper proves the estimated Koopman matrix converges almost surely to the optimal linear Koopman operator for the chosen observables. If true, this means a robot can learn a control policy while it is running, from a small initial dataset, and improve continuously in real time. The paper demonstrates the payoff on a simulated two-link arm and on a soft-actuated hardware platform, where RKL reaches working controllers with a small fraction of the data and training time required by reinforcement learning baselines.","feed_headline":"Online Koopman update equals full retraining, cutting data 10x","feed_subtitle":"Recursive least squares reproduces batch EDMD exactly, so control policies learn in real time from far fewer samples.","key_machinery":"The central object is the finite-dimensional Koopman matrix K, which maps lifted observables (state plus control input) one step forward in the lifted space. The load-bearing identity is the recursive least squares update: with α_k the current lifted snapshot, β_k the next lifted snapshot, and P_k the inverse data covariance, the updates P_{k+1} = P_k − γ_k P_k α_k α_k^T P_k and K_{k+1} = K_k + γ_k (β_k − K_k α_k) α_k^T P_k, with γ_k = 1/(1 + α_k^T P_k α_k), reproduce the batch EDMD solution exactly. This Sherman–Morrison rank-one update carries the entire argument: it makes online updating exact, keeps per-step cost at O(n^2), and lets the convergence proof of EDMD apply to the online estim","core_discovery":"The paper establishes an identity and a convergence theorem. The identity: for a Koopman model estimated via EDMD, appending one new state-action snapshot and applying the recursive least squares update to K and P gives exactly the same K as recomputing the closed-form EDMD solution from the whole dataset. The theorem: if the snapshot sequence is an irreducible, aperiodic, positive-recurrent Markov chain with invariant measure μ, the observables are μ-square-integrable, and the covariance matrices stay full-rank, then the EDMD/RLS estimate converges almost surely to K* = E[φ(f(s))φ(s)^T] (E[φ(s)φ(s)^T])^{-1}, the optimal least-squares Koopman operator in the observation space. The paper also","pith_inferences":["The exact-equivalence identity presupposes that the inverse covariance P stays full-rank. In real deployments, repeated similar states (e.g. a puck lingering in one spot) can make P ill-conditioned, so recursive and batch estimates may drift apart numerically; a practical safeguard would be to periodically re-run batch EDMD and compare.","The convergence proof assumes the data stream is a stationary ergodic Markov chain. In RKL the policy changes at every model update, so the closed-loop transition kernel is time-varying; the paper approximates the trajectories as Markov chains, but a fully rigorous online convergence result would need a law of large numbers for time-inhomogeneous chains.","The same rank-one update argument transfers to any online linear least-squares model, not just Koopman observables; the O(n^2) complexity and exact-equivalence identity should hold for generic linear state-space model updates.","Ergodicity or coverage metrics could be used to actively shape exploration during task execution, steering data collection toward under-sampled regions and avoiding the stagnation failure mode the paper reports; this is a testable extension suggested by the convergence conditions."],"forward_implications":["Online model updates are exact, so no information is ever discarded: the model at every step is the least-squares optimal fit to all accumulated data, not an approximation.","Per-update cost is O(n^2) and independent of dataset size, so model updates can continue at control frequency even in long-horizon, high-rate deployments.","Under the stated ergodic sampling conditions, the learned model converges with probability one to the best possible Koopman operator in the chosen observable space, giving the pipeline a formal consistency guarantee.","Data collected while the controller is attempting the goal is highly informative: the weak ACG hypothesis is empirically supported, and RKL matches or beats RL baselines with under 10% of their data and far less wall-clock time."],"supporting_citations":[{"why":"Supplies the EDMD estimator whose batch solution RKL reproduces with a recursive update.","marker":"[15]"},{"why":"Provides the Sherman-Morrison rank-one inverse update underlying the O(n^2) recursive covariance update.","marker":"[36]"},{"why":"Supplies the strong law of large numbers for Markov chains that yields the almost-sure convergence of EDMD/RLS.","marker":"[32]"},{"why":"Prior online dynamic mode decomposition approach with RLS; presents a similar equivalence result that RKL extends with convergence and complexity analysis.","marker":"[27]"},{"why":"Closest prior Koopman-model MPC with recursive online updates; RKL differs by working in observable space and removing the termination-condition obstruction.","marker":"[28]"},{"why":"Koopman-based control formulation and MPC-SAC derivation used in the pipeline; also demonstrates standard normal-distribution initialization issues.","marker":"[21]"},{"why":"Provides the DMD-with-control formulation K_z, K_g used to build the control-affine Koopman model.","marker":"[35]"},{"why":"Supplies the Sequential Action Control solver used as the MPC policy in experiments.","marker":"[37]"},{"why":"Baseline Koopman model-based control (KL) that RKL is compared against in the experiments.","marker":"[41]"}],"fun_headline_variants":["Recursive Koopman matches batch EDMD exactly","Online Koopman update equals retraining, 10x less data","Provable convergence for real-time Koopman learning","Sample-efficient control via exact online Koopman updates","Koopman RLS: same as batch, but real-time and light"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The data stream must behave like a random process with a fixed long-run distribution over states and actions; in the actual online loop the controller changes after every model update, so this stationarity is not guaranteed, and if it fails the convergence proof does not cover the RKL process.","fun_headline_variants_meta":{"raw":{"variants":["Recursive Koopman matches batch EDMD exactly","Online Koopman update equals retraining, 10x less data","Provable convergence for real-time Koopman learning","Sample-efficient control via exact online Koopman updates","Koopman RLS: same as batch, but real-time and light"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1695,"prompt_tokens":768,"completion_tokens":927,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":842}},"tokens_in":512,"tokens_out":927,"duration_ms":10954,"temperature":1.0,"reasoning_tokens":842,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:57:45.980089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed recorded data stream, compute the batch EDMD Koopman matrix once, then feed the same snapshots one by one through the RLS updates and compare the final matrices; the paper's identity predicts equality up to floating-point roundoff. A concrete way to observe failure: if the inverse covariance P becomes ill-conditioned after many similar states, recursive and batch estimates will diverge. Separately, during an RKL run on hardware, record the empirical distribution of (state, action) pairs over time; if that distribution keeps shifting as the policy updates rather than settling down,","supporting_citations":[],"review_version":1}