{"id":"f408baf8-79ab-4f49-b34d-5740905315c4","arxiv_id":"2505.24572","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A SARSA reinforcement learning agent is trained offline and deployed online to adaptively set the DeePC regularization hyperparameter, with simulations showing competitive or better tracking under Gaussian and uniform noise.","lead":"This paper trains a reinforcement learning agent to automatically adjust the single most important tuning dial of a data-driven controller (DeePC) while the system runs under changing noise. In simulations on a noisy second-order system and a triple-mass spring, the adaptive version converges and recovers from noise shifts at least as well as the compared tuning methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"S-DeePC's central claim of near-optimal λ_g tracking is unverified: the paper never compares the λ_g chosen online with the oracle optimal λ_g, and the assumed Markov state is not established. An oracle comparison would settle whether the method truly tracks the optimum.","rationale":"The paper proposes a plausible engineering solution: using SARSA to adapt λ_g in DeePC. The simulations show moderate improvements in the composite objective J over fixed λ_g, Hanke, and DeePC-Hunt in several scenarios. My stress-test focuses on the central claim as stated in the abstract: 'the results demonstrate the identification of near-optimal hyperparameters.' For this claim to hold, the λ_g chosen online must be close to the optimal λ_g, and the method must adapt it when noise statistics change. The paper never reports λ_g, so this is not established. The reader's weakest assumption—that x_k=(||u||, M) is a Markov state and the offline-trained Q-table transfers—is a plausible theoretical weakness, but it is secondary to the missing oracle comparison: even if the state were Markov, the paper would still need to show near-optimal selection. The proposed test—reporting the chosen λ_g versus an oracle—settles the central claim directly. I therefore maintain the CONDITIONAL verdict: the authors should supply this comparison, the code, and repeated trials. I do not see a demonstrated falsehood, only an unverified central assertion, so no stronger verdict is warranted.","tokens_in":15092,"tokens_out":5500,"duration_ms":63786,"concrete_test":"In the Gaussian experiment (Section IV-B2), log the discrete λ_g selected by S-DeePC at each time step. Separately, for the same noise realizations, compute the oracle optimal λ_g by exhaustively solving (9) over the grid {0.006:0.006:0.606} with the true noise distribution to minimize expected J. Plot S-DeePC's selected λ_g against the oracle optimum, both before and after the noise change at t=6s. If the median absolute deviation exceeds 0.05 or the selected values do not shift toward the new optimum within the settling window, the near-optimal tracking claim is refuted. Repeat over 20 random seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that S-DeePC identifies near-optimal DeePC hyperparameters λ_g and adapts them when noise changes. Yet Section IV never reports the λ_g values selected by the algorithm; it plots only the composite objective J of Eq. (8). Without measuring the gap between the chosen λ_g and the true optimal λ_g for a given noise level, the phrase 'near-optimal' is unsupported. The RL state in Algorithm 1 and Eqs. (10e)-(10g) is x_k=(||u||_{2,n}, M_k), and the paper does not demonstrate that this two-scalar state is a sufficient statistic for the optimal λ_g; the transition model Eq. (11) is treated as unknown but the Bellman recursion (17)-(18) requires Markovianity. The transfer from offline training to online operation with different noise levels is also assumed rather than tested. If the state is not Markov, or if the selected λ_g is far from optimal but still improves on fixed baselines, the central claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S-DeePC, a reinforcement-learning-based adaptive tuning scheme for the regularization parameter λ_g in data-enabled predictive control (DeePC). The method trains a SARSA Q-table offline and then adjusts λ_g online using a two-dimensional state (the control input norm and the root-mean-square output error), aiming to maintain performance when noise conditions change without prior system knowledge. The authors claim near-optimal hyperparameter identification and robustness, supported by numerical simulations on a second-order LTI system and a triple-mass-spring system, with comparisons against baseline DeePC, Hanke, and DeePC-Hunt. The central claims are that the learned policy tracks the optimal λ_g and that the offline-trained model transfers to online operation under varying noise.","tokens_in":15337,"tokens_out":6754,"duration_ms":75373,"significance":"If the central claims held, the paper would offer a practical, low-computation way to replace manual or exhaustive DeePC hyperparameter tuning and to adapt to changing noise conditions online. The offline/online decomposition and the use of a Q-table for λ_g selection are sensible engineering directions, and the comparison with Hanke and DeePC-Hunt addresses a relevant literature. However, the theoretical derivation contains mathematical errors, the empirical evaluation does not report the selected λ_g values or compare them to an oracle optimum, and the hyperparameters of the RL scheme are tuned on the same test scenario used for evaluation. These issues mean that the current evidence does not establish near-optimal λ_g tracking or the claimed robustness; the paper's contribution is therefore not yet substantiated, though the underlying idea is salvageable.","major_comments":[{"comment":"The central claim that S-DeePC identifies 'near-optimal' λ_g is never directly tested. The paper plots the composite objective J of Eq. (8) but never reports the λ_g values chosen online or their gap to the oracle optimal λ_g for each noise condition. An improvement in J over fixed baselines does not establish near-optimal hyperparameter tracking. Please add a plot of the online-selected λ_g(t) versus the optimal λ_g found by exhaustive search for each noise level, and quantify the gap.","section":"Section IV-B and Abstract/Conclusion"},{"comment":"The Bellman recursion contains an unjustified interchange of max and expectation. In Eq. (18), the third equality rewrites max over future policies of an expectation as the expectation of a max, which is not valid in general; the max over future policies should be conditional on the realized next state, not taken outside the expectation. In addition, Eq. (15) defines Q_{z→c}(x, λ_g) with an argmax over λ_g while also taking λ_g as an argument, which is circular. These issues undermine the theoretical foundation for the Q-learning/SARSA update in Eq. (20).","section":"Section III, Eqs. (15)-(18)"},{"comment":"The stochastic optimization problem (10a)-(10g) introduces probability densities pf_{x,k}, pf_k, pf_{λ,k} and variables λ̂_{g,k} that are never defined or used in the rest of the paper. The paragraph after Eq. (10) also refers to functions f_{u,k}, μ_k, and û_k that are undefined. This makes the claimed 'probabilistic inference' step vacuous, and the connection between Eq. (10) and the SARSA algorithm is not established.","section":"Section III, Eq. (10)"},{"comment":"The RL meta-parameters and horizon n are selected by minimizing the same objective J on the same experimental setup later used for performance evaluation (Figs. 5-6). This in-sample tuning means the reported superiority of S-DeePC may partly reflect overfitting to the test scenario. Please use a separate validation set, or report performance across a range of hyperparameters without selecting on the test data.","section":"Section IV-A"},{"comment":"The state x_k = (||u||_{2,n}, M_k) together with λ_{g,k} is assumed to be a Markov state for the controlled system, and the offline-trained Q-table is assumed to transfer to online operation with different noise levels. Neither assumption is verified. The transition model (11) is unknown, and the Bellman recursion (17) requires Markovianity; the paper should test the validity of the state representation, for example by checking whether the learned Q-values yield consistent λ_g choices for the same state under different noise histories.","section":"Section III, Eqs. (10e)-(11) and Algorithm 1"}],"minor_comments":[{"comment":"The text refers to 'baseline PeePC'; this should read 'baseline DeePC.'","section":"Section IV, introductory paragraph"},{"comment":"The sentence 'We would like to not that random additive noise...' contains a typo; it should read 'note.'","section":"Section III, paragraph after Eq. (9)"},{"comment":"The DeePC formulation in Eq. (6) is not the standard formulation used in the simulations; the paper does not explain how this bilevel rank-constrained problem is implemented in MATLAB/OSQP. Please clarify the actual optimization problem solved.","section":"Section II, Eq. (6)"},{"comment":"The notation ||e||_n is ambiguous: it is defined as sqrt(e^T n e) where n is a matrix, but n is also used as the number of samples later. Use a different symbol for the weighting matrix.","section":"Section II, Notation"},{"comment":"The paper calls the method SARSA, but Eqs. (17)-(18) are Q-learning Bellman optimality equations, while Eq. (20) is the SARSA update rule. Please clarify the relationship between the two and which update is actually used.","section":"Section III, Eqs. (17)-(20)"},{"comment":"The lines 'Hankel matrix←(3)' and 'λ_g,k, x_k ← Discretize U_p, Y_p, λ_g, using (11)' are vague; specify the discretization procedure and how Eq. (11) is used to construct the state and action.","section":"Algorithm 1"},{"comment":"Reference [46] is cited for the triple-mass-spring system, but [46] is a paper on the relationship between DeePC and subspace predictive control and does not describe that system; the reference appears to be incorrect.","section":"Section IV-C"},{"comment":"The conclusion states that the method 'can effectively identify optimal DeePC hyperparameter', but no selected λ_g values are reported; please align the conclusion with the available evidence.","section":"Section V, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the offline/online RL tuning idea is plausible. However, the central claim of near-optimal λ_g tracking is unsupported by the reported experiments, the Bellman recursion contains mathematical errors, and the hyperparameter selection is in-sample. These are load-bearing issues, but they are fixable with additional experiments and a corrected derivation; hence I recommend major revision rather than rejection. I would also note that the paper's citation list contains several references that do not match the claims (e.g., [46] for the spring system), which should be checked carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible engineering idea—train a SARSA agent offline to pick DeePC's regularization parameter λ_g online—and the simulations show it can beat fixed-λ DeePC, Hanke, and DeePC-Hunt in several noisy regimes. But the paper's central claim that it identifies 'near-optimal' λ_g is not demonstrated: the authors never report the λ_g values their algorithm chose or compare them to the true optimal λ_g. The theory section is mostly unfilled scaffolding; it should be fixed or cut.\n\nWhat's genuinely new: applying SARSA to DeePC hyperparameter tuning, with offline Q-table pre-training and online adaptation. I haven't seen that exact combination in the cited literature. The paper also includes two adaptive baselines (Hanke, DeePC-Hunt) and a second physical system (triple-mass-spring), which is more than many papers in this space do.\n\nNow the soft spots, in rough order of severity.\n\nFirst, the 'near-optimal' claim. The abstract says the method identifies near-optimal hyperparameters, but Section IV plots only the composite objective J. We never see whether λ_g actually tracks the optimal λ_g when the noise doubles at t=6s or t=15s. Without that oracle comparison, the claim is unsupported. The method could be picking mediocre λ_g values and still beat fixed baselines—useful, but weaker than claimed.\n\nSecond, the state representation. The RL state is x_k=(||u||_{2,n}, M_k), two scalars. Q-learning needs a Markov state, and the paper never argues why these two are sufficient statistics for the optimal λ_g. The Bellman recursion in (17)-(18) also exchanges expectation and max without justification, and the conditional densities introduced in (10) are never derived or used. These are addressable issues, but as written a reviewer will rightly flag them.\n\nThird, the evaluation is in-sample. The RL meta-parameters (α, γ, ε) are tuned to minimize J on the same setup used for the comparison runs, and all plots are single runs with no error bars. I'd want repeated trials with different noise seeds and at least a short sensitivity analysis.\n\nThere's no code or data release, which makes reproducibility harder. The spring-control demo also omits the fixed-λ DeePC baseline, so we can't see if the adaptive methods actually matter there.\n\nWho is this for? Control engineers working on practical DeePC tuning who want an off-the-shelf adaptive λ_g. The idea deserves a serious look. I'd send it to peer review—it's not a desk-reject—but I'd insist on major revision: report λ_g trajectories with oracle comparison, fix or cut the theory, add repeated trials, and release code. With those changes, it could be a solid applied paper.","headline":"Plausible SARSA-based λ_g tuner for DeePC with useful simulations, but the 'near-optimal' claim is unverified and the theory section needs a rewrite.","tokens_in":15854,"tokens_out":3302,"would_cite":false,"duration_ms":37641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93C40","93E20","93C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Q-table trained offline can select near-optimal DeePC regularization hyperparameters online, letting data-enabled predictive control stay effective as noise changes.","keywords":["data-enabled predictive control","hyperparameter tuning","reinforcement learning","sequential decision-making","SARSA","DeePC","noise robustness"],"falsifier":"A direct test: train the Q-table offline with Gaussian noise of variance $10^{-6}$, then run S-DeePC online while the noise variance is ramped continuously from $10^{-6}$ to $10^{-2}$, well outside the training range. If the Markov-state and transfer assumptions hold, tracking error $M$ should stay near its minimum; if the method's adaptivity is only interpolation within the training distribution, $M$ should rise sharply and the selected $\\lambda_g$ should fail to track the optimal value. Running the same experiment on a different plant with the same training data would test whether the table encodes plant-specific behavior.","tokens_in":14817,"feed_emoji":"🎛️","tokens_out":4890,"duration_ms":53839,"temperature":0.7,"pith_summary":"The paper proposes S-DeePC, a version of data-enabled predictive control (DeePC) whose regularization parameter $\\lambda_g$ is adjusted in real time by a reinforcement learning agent rather than fixed in advance or tuned by exhaustive search. The authors argue that the optimal $\\lambda_g$ for a noisy system drifts as noise and operating conditions change, and that a Q-table trained offline on input–output data can be used online to select near-optimal $\\lambda_g$ at each step. They model the tuning problem as a sequential decision problem whose state is the pair $(\\|u\\|_{2,n}, M)$, where the first is control energy and $M$ is root-mean-square tracking error, and whose reward is the negative of a weighted difference of the two. In numerical comparisons against fixed-parameter DeePC, the Hanke rule, and DeePC-Hunt, the proposed method adapts faster to a sudden doubling of Gaussian or uniform noise and stabilizes with competitive tracking performance.","feed_headline":"Reinforcement learning tunes DeePC's lambda in real time","feed_subtitle":"S-DeePC adapts its regularization parameter when noise changes, keeping performance stable without system identification.","key_machinery":"The load-bearing object is the Q-table learned by SARSA, with state $x_k = (\\|u\\|_{2,n}, M_k)$, action $\\lambda_g$, and reward $R_k = -|a x_k(1) - b x_k(2)|$. The authors justify this choice by an empirical curve (Fig. 2) linking $\\lambda_g$ to input energy and tracking error, which exhibits a critical inflection point $C$. The Bellman-style recursion (Eq. 17) and the update rule $Q^{\\text{new}} \\leftarrow Q^{\\text{old}} + \\alpha(R + \\gamma Q^{\\text{old}}(\\text{next}) - Q^{\\text{old}})$ (Eq. 20) carry the learning; the online stage loads the trained Q-table and switches $\\lambda_g$ with probability $1-\\varepsilon$ to the argmax action, with $\\varepsilon$ allowing exploration. The offline/online split keeps the online computation light enough for real-time control.","core_discovery":"The central discovery, as the authors present it, is that a DeePC controller's regularization hyperparameter $\\lambda_g$ can be mapped to measurable I/O quantities—control input energy $\\|u\\|_{2,n}$ and output error $M$—and that this mapping has a consistent shape: increasing $\\lambda_g$ first sharply reduces input energy, then saturates, while output error $M$ dips slightly and then rises. This empirical relationship lets the tuning problem be recast as a Markov decision process, where the state $x_k = (\\|u\\|_{2,n}, M_k)$, the action is the next $\\lambda_g$, and the reward is $R_k = -|a\\|u\\|_{2,n} - bM|$. The authors show that SARSA with an $\\varepsilon$-greedy policy, trained offline on noisy trajectories and deployed online, updates a Q-table via the standard recursion and selects $\\lambda_g$ that tracks the near-optimal value when noise intensity changes. The claimed result is not a theorem but a demonstrated mechanism: an offline-trained table gives real-time adaptivity without system identification or prior knowledge of the noise distribution.","pith_inferences":["A testable extension: train the Q-table on one plant and run it on a second plant with different dynamics; the paper's claim of operating without prior knowledge predicts transfer, but the Markov assumption would likely break, causing degraded performance.","The reward $J = |a\\|u\\| - bM|$ with $a=b=1$ fixes the trade-off at one point; the paper leaves open how to set $a$ and $b$ systematically when the control objective is strict output tracking with a hard input constraint.","The empirical $\\lambda_g$-to-I/O curve has a threshold structure, so an agent that learns which side of the inflection point it is on could be replaced by a simple switching rule; the paper does not test whether the Q-table actually encodes that structure.","Because the Q-table is trained on a fixed noise distribution, the claimed robustness across noise levels requires the table to generalize beyond its training distribution, which is untested."],"forward_implications":["If the method works as claimed, DeePC can be deployed with a pre-trained Q-table and maintain performance when noise statistics change, without re-solving an optimization or re-identifying the system.","The tuning framework is not tied to DeePC's specifics; any controller whose performance depends on a scalar regularization weight in a similar I/O-responsive way could use the same state-action-reward design.","Because the expensive learning happens offline, the online controller needs only table look-ups and the DeePC update, avoiding the overhead of methods that re-optimize $\\lambda_g$ online.","The state is built from quantities already computed during control, so the method needs no extra sensors or models.","The approach is demonstrated to regain stability within about ten seconds after a noise doubling, suggesting a practical adaptation timescale for slowly varying disturbances."],"supporting_citations":[{"why":"Establishes that $\\lambda_g$ plays a critical role in DeePC's robustness, motivating the need for adaptive tuning.","marker":"[42]"},{"why":"Shows that maintaining a fixed $\\lambda_g$ can degrade control performance as data volume increases, providing the core motivation for real-time adjustment.","marker":"[43]"},{"why":"Argues that real-world DeePC implementations demand a rigorous tuning methodology, framing the problem the paper addresses.","marker":"[44]"},{"why":"Presents a prior real-time tuning method based on dynamic penalties, which the paper identifies as relying on prior system knowledge.","marker":"[52]"},{"why":"Supplies the Hanke-Raus rule baseline method that the paper compares against in simulations.","marker":"[53]"},{"why":"Supplies the DeePC-Hunt differentiable-optimization baseline that the paper compares against in simulations.","marker":"[55]"},{"why":"Provides the Q-learning and SARSA machinery used to build the reinforcement learning tuner.","marker":"[56]"},{"why":"Provides the standard reinforcement learning background, including $\\varepsilon$-greedy exploration, that the SARSA update relies on.","marker":"[57]"}],"fun_headline_variants":["RL adapts DeePC's regularization on the fly","SARSA tunes DeePC's lambda for noisy systems","Adaptive DeePC via RL: no system ID needed","Real-time lambda tuning for DeePC without system ID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the pair (input energy, tracking error) together with the current $\\lambda_g$ is a sufficient state for choosing the next $\\lambda_g$, and that a Q-table trained offline on one noise level still selects good actions when noise changes.","fun_headline_variants_meta":{"raw":{"variants":["RL adapts DeePC's regularization on the fly","SARSA tunes DeePC's lambda for noisy systems","Adaptive DeePC via RL: no system ID needed","Real-time lambda tuning for DeePC without system ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2854,"prompt_tokens":936,"completion_tokens":1918,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":552,"tokens_out":1918,"duration_ms":15381,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:18:52.387266+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: train the Q-table offline with Gaussian noise of variance $10^{-6}$, then run S-DeePC online while the noise variance is ramped continuously from $10^{-6}$ to $10^{-2}$, well outside the training range. If the Markov-state and transfer assumptions hold, tracking error $M$ should stay near its minimum; if the method's adaptivity is only interpolation within the training distribution, $M$ should rise sharply and the selected $\\lambda_g$ should fail to track the optimal value. Running the same experiment on a different plant with the same training data would test whether the table encodes plant-specific behavior.","supporting_citations":[{"cited_title":"Robust data-enabled predictive control: Tractable formulations and performance guarantees.IEEE Transactions on Automatic Control, 68(5):3163–3170, 2023","cited_arxiv_id":null,"evidence_quote":"Establishes that $\\lambda_g$ plays a critical role in DeePC's robustness, motivating the need for adaptive tuning."},{"cited_title":"On the equivalence of direct and indirect data-driven predictive control approaches.IEEE Control Systems Letters, 2024","cited_arxiv_id":null,"evidence_quote":"Shows that maintaining a fixed $\\lambda_g$ can degrade control performance as data volume increases, providing the core motivation for real-time adjustment."},{"cited_title":"Data-driven predictive control with improved per- formance using segmented trajectories.IEEE Transactions on Control Systems Technology, 31(3):1355–1365, 2022","cited_arxiv_id":null,"evidence_quote":"Argues that real-world DeePC implementations demand a rigorous tuning methodology, framing the problem the paper addresses."},{"cited_title":"Harnessing uncertainty for a separation principle in direct data- driven predictive control.Automatica, 173:112070, 2025","cited_arxiv_id":null,"evidence_quote":"Presents a prior real-time tuning method based on dynamic penalties, which the paper identifies as relying on prior system knowledge."},{"cited_title":"Offset–free data–driven predictive control","cited_arxiv_id":null,"evidence_quote":"Supplies the Hanke-Raus rule baseline method that the paper compares against in simulations."},{"cited_title":"DeePC-Hunt: Data-enabled Predictive Control Hyperparameter Tuning via Differentiable Optimization","cited_arxiv_id":"2412.06481","evidence_quote":"Supplies the DeePC-Hunt differentiable-optimization baseline that the paper compares against in simulations."},{"cited_title":"Q- learning based linear quadratic regulator with balanced exploration and exploitation for unknown systems","cited_arxiv_id":null,"evidence_quote":"Provides the Q-learning and SARSA machinery used to build the reinforcement learning tuner."},{"cited_title":"Reinforcement learning: An introduction","cited_arxiv_id":null,"evidence_quote":"Provides the standard reinforcement learning background, including $\\varepsilon$-greedy exploration, that the SARSA update relies on."}],"review_version":1}