{"id":"92eea322-030d-4a06-81c5-169447025a8d","arxiv_id":"2412.17012","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A data-driven adaptive controller for a class of positive linear systems is derived from a Q-learning Bellman equation, with robustness bounds and a stochastic shortest path comparison.","lead":"The paper designs an online adaptive controller for a class of positive linear systems, using data to solve a Q-learning equation without needing the system model. It proves bounds on the cost and robustness to noise, and tests the method on stochastic shortest path problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's lower-bound constant alpha_hat is non-positive for rho*beta in [1/2,1), but the proof divides by it; assumption rho*beta<1 is insufficient.","rationale":"The central contribution of the paper is a sufficient robustness condition for the adaptive data-driven policy. The proof of Theorem 1 is the linchpin: it provides the two-sided bound (20) that Theorem 2 and Corollary 1 rely on. The reader's weakest_assumption identifies exactly the load-bearing flaw: alpha_hat = 1 - rho*beta/(1 - rho*beta) must be positive for the argument to work, but the theorem only assumes rho*beta < 1. I checked Appendix A.3: the proof divides both sides of an inequality by alpha_hat when deriving p <= alpha_hat^{-1}*p(t). For rho*beta in [1/2,1), alpha_hat <= 0, so the division is invalid or reverses the inequality, and the claimed bound does not follow. This is an internal inconsistency in the theorem's hypotheses, not merely a question of conservatism or a disagreement with existing consensus. The issue propagates: Theorem 2 and Corollary 1 both depend on (20), so the advertised stability and regret guarantees are not established for the stated parameter range. I see no need to manufacture an additional concern; this one is decisive. I also note the paper does contain useful independent components: Lemma 1 gives an algebraic equivalence between q-iteration and p-iteration, and the data-driven equation construction in Section 2 is a reasonable formulation. But the main theorem's proof has a load-bearing gap. A repair is plausible by strengthening the assumption to rho*beta < 1/2 and reworking the constants, but as written the central claim is not proven. Since the reader already recommended REJECT and my analysis agrees, the verdict should remain unchanged.","tokens_in":15286,"tokens_out":4338,"duration_ms":39034,"concrete_test":"Set rho = 0.75, beta = 1 (so rho*beta = 0.75 < 1) and inspect the step in Appendix A.3 following the definition of alpha_hat. Compute alpha_check = 0.25 and alpha_hat = 1 - 0.75/0.25 = -2. The text divides both sides of an inequality by alpha_hat; because the divisor is negative, the inequality direction must reverse, so the subsequent application of Lemma 2 is invalid. If the authors instead add the condition rho*beta < 1/2 to Theorem 1 and re-derive the bounds with alpha_hat > 0, the proof becomes consistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 (Section 3.1) states a two-sided bound with constants alpha_check = 1 - rho*beta and alpha_hat = 1 - alpha_check^{-1}*rho*beta, and the proof in Appendix A.3 divides by alpha_hat to conclude p <= alpha_hat^{-1}*p(t). But alpha_hat = 1 - rho*beta/(1 - rho*beta) is positive only if rho*beta < 1/2. Under the stated assumption rho*beta < 1, any value in [1/2,1) makes alpha_hat <= 0. For example, rho*beta = 3/4 gives alpha_hat = -2; dividing by -2 reverses the inequality, so the claimed derivation collapses and the theorem as stated is not established. The lower-bound direction may be vacuously true with alpha_hat < 0, but the proof's logic fails precisely at the load-bearing step, and Theorem 2/Corollary 1 inherit the invalid bound. A secondary issue is that (19) is required for all t >= t0, which is a persistence/adequacy condition; however, the alpha_hat gap is sufficient by itself to invalidate the stated theorem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive, data-driven controller for a class of positive linear systems with linear costs, based on a 'data-driven algebraic equation' derived from the Q-function Bellman equation. The main results (Theorem 1, Theorem 2, Corollary 1) aim to establish that, under a small-model-mismatch condition (19) and rho*beta < 1, the adaptive policy's value function p(t) remains within constant factors of the optimal p, leading to a suboptimality bound on cumulative cost. A numerical experiment compares the policy with Q-learning on a stochastic shortest path example.","tokens_in":15549,"tokens_out":9442,"duration_ms":83587,"significance":"If established, the result would give a simple online scheme with explicit robustness margins and a quantitative excitation-robustness trade-off for an important class of positive systems, with a reproducible numerical study. The connection to SSP and the data-driven equation are interesting. However, the main theorem contains a load-bearing technical error that invalidates the stated guarantees; the contribution is therefore not yet supported.","major_comments":[{"comment":"The constant alpha_hat = 1 - rho*beta/alpha_check, with alpha_check = 1 - rho*beta, is positive only if rho*beta < 1/2. The theorem states the result for all rho*beta < 1, but the proof divides by alpha_hat to obtain p <= alpha_hat^{-1} p(t). For rho*beta in [1/2, 1), alpha_hat <= 0, so the division reverses the inequality and the two-sided bound (20) is not derived. For example, rho*beta = 3/4 gives alpha_hat = -2. Since Theorem 2 and Corollary 1 rely on (20), the main stability and cost guarantees are not established under the stated assumptions. The theorem should be restated with the condition rho*beta < 1/2, or the proof must be revised, and the downstream results adjusted accordingly.","section":"Section 3.1, Theorem 1 and Appendix A.3"},{"comment":"Condition (19) is assumed to hold for all t >= t0, which is a persistence-of-excitation and small-disturbance condition rather than a consequence of the proposed algorithm. Remark 5 acknowledges this, but the main theorems do not provide any online verifiable certificate for (19). Since the stability guarantee is conditional on this assumption, the paper should state more explicitly that (19) is an additional hypothesis on the exploration signal and the disturbance bound relative to the state, and it should discuss how a practitioner could check it online or what happens when it fails.","section":"Section 3.1, condition (19)"}],"minor_comments":[{"comment":"The displayed transition matrices T(1)-T(4) and cost vectors c(1)-c(4) have inconsistent bracket placement and row lengths, making the SSP instance difficult to verify from the text.","section":"Section 4"},{"comment":"The chain s <= p <= (beta min_i s_i) 1 <= beta s is unclear: the middle term is a scalar times the all-ones vector, and the final inequality uses elementwise order. Please clarify the intended ordering.","section":"Definition 1"},{"comment":"The displayed bound would be more readable with explicit parentheses as gamma^{-1}( p^T x_{t0} + sum ... ); as written, the scope of gamma^{-1} is ambiguous.","section":"Corollary 1"},{"comment":"The definition of min{A,0} as 'the minimum element of A' conflicts with its later use as an elementwise operator in equations such as (3) and (5).","section":"Notation, Section 1.2"},{"comment":"The passage from (9) to (10) should justify why the min over K can be moved inside the row of stacked states; the validity relies on the minimizing K being the same for each column, which holds for the true q-parameter but should be stated explicitly.","section":"Section 2.2, equations (10)-(12)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is built directly on the authors' previous work (Ohlin et al., 2024a,b; Bencherki and Rantzer, 2024) and the novelty is incremental. The primary issue is the technical error in Theorem 1, which appears fixable by adding the condition rho*beta < 1/2 and adjusting the downstream statements. If the authors can do that and clarify condition (19), the paper could be suitable for publication. The 'model-free' framing is somewhat overstated, since the data-driven equation still relies on estimating A and B through \\bar{Sigma} Sigma^{-1}."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I agree with the reader's verdict. The paper's main idea is genuinely new: it turns the Q-learning Bellman equation for positive linear systems into a data-driven algebraic equation driven by correlation matrices that don't grow with time, and then uses it to update the feedback gain online without identification. The numerical SSP comparison is a nice touch and the code is available. That part is the real contribution and it is worth taking seriously.\n\nThe soft spot is exactly where the reader puts it. Theorem 1 claims two-sided bounds with constants alpha_hat = 1 - alpha_check^{-1}*rho*beta and alpha_check = 1 - rho*beta, under rho*beta < 1. But alpha_hat is positive only when rho*beta < 1/2. The proof in Appendix A.3 divides by alpha_hat to get p <= alpha_hat^{-1} p(t); if alpha_hat is negative the inequality flips and the conclusion doesn't follow. So the lower-bound direction is not established for the stated range. Since Theorem 2 and Corollary 1 use the same constants, all the performance/robustness guarantees inherit the problem. That's a load-bearing flaw, not a typo.\n\nI'd also note that condition (19) for all t >= t0 is a strong persistence-of-excitation and disturbance-bound assumption, but that is secondary; the alpha_hat issue alone invalidates the theorem as stated.\n\nThe good news is the flaw looks fixable: strengthen the assumption to rho*beta < 1/2, and rework the lower-bound argument, or replace the two-sided bound with a one-sided upper bound plus a separate lower bound under different conditions. The derivation before Theorem 1 is sound and the numerical performance is plausible.\n\nMy recommendation: send it to review with a clear request for major revision. The idea deserves referee time, but the theorem as written can't stand. I would not cite the current version for the stability guarantee, but I'd keep an eye on a corrected version.","headline":"Novel data-driven Q-learning setup for positive systems, but Theorem 1's lower-bound constant can be non-positive under the stated assumption, so the main robustness guarantee is unproven as written.","tokens_in":16033,"tokens_out":2300,"would_cite":false,"duration_ms":20101,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93C40","93D09","93C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-driven controller for positive systems keeps the value vector within explicit bounds.","keywords":["adaptive control","positive systems","data-driven control","Q-learning","Stochastic Shortest Path","robust control","model-free Bellman equation","linear cost networks"],"falsifier":"Evaluate the constants at a pair with $\\rho\\beta\\in(1/2,1)$, for example $\\beta=1.5$ and $\\rho=0.4$: then $\\hat{\\alpha}=1-0.6/0.4=-0.5$, so the claimed lower bound $\\hat{\\alpha}p\\leq p(t)$ is vacuous and the proof's division by $\\hat{\\alpha}$ is illegitimate. A concrete check would simulate a plant in $\\mathcal{M}_\\beta$ with a disturbance sequence satisfying (19) at that $\\rho$ and test whether $p(t)$ still satisfies a non-vacuous two-sided bound; if it does, the theorem's range can be repaired, and if not, the stated range must be restricted.","tokens_in":15075,"feed_emoji":"⚙️","tokens_out":7223,"duration_ms":57762,"temperature":0.7,"pith_summary":"This paper proposes an adaptive controller for positive linear systems, systems whose state stays nonnegative like buffers, queues, or probabilities, solving an infinite-horizon optimal control problem online without first identifying the dynamics. The controller updates a Q-factor from data correlation matrices with a forgetting factor and solves a data-driven algebraic equation that is the model-free form of the Bellman equation. The central result, Theorem 1, shows that if the estimated model's mismatch to the true plant stays below an explicit bound measured by $\\rho$, and if $\\rho$ times a stabilizability constant $\\beta$ is small, then the data-driven value vector $p(t)$ stays within multiplicative factors of the true optimal value vector $p$; Corollary 1 turns this into a finite bound on cumulative cost. Because the problem class is equivalent to Stochastic Shortest Path, the same policy can be compared with model-free SSP algorithms and is shown numerically to achieve lower regret. The value of the result is an explicit, quantitative trade-off between excitation and robustness to unmodeled dynamics.","feed_headline":"Data-driven controller for positive systems gets stability guarantee","feed_subtitle":"Explicit condition links excitation to robustness and bounds the cumulative cost","key_machinery":"The central objects are the data correlation matrices $\\Sigma(t)$ and $\\bar{\\Sigma}(t)$ with forgetting factor $\\lambda$: $\\Sigma$ accumulates weighted outer products of past state-input pairs, and $\\bar{\\Sigma}$ accumulates weighted products of the next state with the current state-input pair. Their ratio $\\bar{\\Sigma}(t)\\Sigma^{-1}(t)$ is an online estimate $(\\hat{A}(t),\\hat{B}(t))$ of the dynamics, and the difference from $(A,B)$ is $\\tilde{\\Sigma}(t)\\Sigma^{-1}(t)$, the model mismatch controlled by condition (19). The data-driven algebraic equation in the $q$-parameter is the model-free Bellman equation in Q-factor form, and Lemma 1 shows that value iteration on $q$ is algebraically equivalent to value iteration on $p$, so solving (12) is a direct substitute for knowing $(A,B)$. The proof machinery is perturbation analysis of the algebraic equation (3)/(17) together with the comparison lemma, Lemma 2, which says that any supersolution of the Bellman equation gives an upper bound on the optimal value.","core_discovery":"The paper claims that a policy extracted from the solution $q(t)$ of the data-driven algebraic equation $\\left(q(t)-\\begin{bmatrix}s\\\\r\\end{bmatrix}\\right)^\\top \\Sigma(t)=\\min_{K(t)\\in\\mathcal{K}(E)} q(t)^\\top \\begin{bmatrix}I\\\\ K(t)\\end{bmatrix} \\bar{\\Sigma}(t)$ stabilizes every plant in the model set $\\mathcal{M}_\\beta$, provided the perturbation bound (19) holds with $\\rho\\beta<1$. Under that condition, the associated value vector $p(t)=\\begin{bmatrix}I & K^\\top(t)\\end{bmatrix}q(t)$ satisfies $\\hat{\\alpha}p\\leq p(t)\\leq \\check{\\alpha}^{-1}p$ with $\\check{\\alpha}=1-\\rho\\beta$ and $\\hat{\\alpha}=1-\\check{\\alpha}^{-1}\\rho\\beta$. This two-sided inequality is the paper's core: it says the adaptive estimate never drifts too far from the true optimal cost vector, and it yields the cumulative-cost bound of Corollary 1, in which a factor $\\gamma\\to 1$ as $\\rho\\to 0$ closes the suboptimality gap. The argument works through a perturbation analysis of the algebraic equation, comparing the true plant $(A,B)$ with the estimated pair $(\\hat{A}(t),\\hat{B}(t))=\\bar{\\Sigma}(t)\\Sigma^{-1}(t)$.","pith_inferences":["Condition (19) is computable online from $\\Sigma(t)$ and $\\bar{\\Sigma}(t)$, so the controller could in principle monitor its own validity and increase exploration when the mismatch bound is approached; the paper does not develop this adaptation.","The proof's division by $\\hat{\\alpha}$ indicates that the meaningful robustness margin is $\\rho\\beta<1/2$, not $\\rho\\beta<1$ as stated in Theorem 1; sharpening or fixing this gap is a natural next step.","Because the algebraic equation is solved by value iteration or a linear program, the same framework could extend to other structured MDPs whose Bellman equation has an affine solution, such as average-cost positive systems or processing-network dispatching.","The regret comparison used $\\epsilon$-decreasing exploration; equipping the policy with more efficient exploration strategies from the SSP literature is an untested but plausible improvement."],"forward_implications":["If condition (19) holds after some time $t_0$, the adaptive policy stabilizes the closed loop in the sense that the true value vector $p(t)$ remains between $\\hat{\\alpha}p$ and $\\check{\\alpha}^{-1}p$, so the controller can be used online without an identification step.","The bound quantifies the excitation-robustness trade-off: plants that are harder to stabilize, corresponding to larger $\\beta$, require smaller mismatch $\\rho$, so more or better excitation must be injected.","As the mismatch $\\rho$ tends to zero, the constants $\\check{\\alpha}$ and $\\hat{\\alpha}$ tend to $1$, and the suboptimality gap in Corollary 1 closes, recovering the optimal policy.","Corollary 1 gives a finite-time cumulative cost bound in which only the exploration noise and disturbances appear additively, so regret is controlled by the excitation injected and the disturbance magnitude.","For Stochastic Shortest Path instances, the same guarantees transfer via the equivalence of problem classes, and the numerical comparison indicates lower regret than $\\epsilon$-decreasing Q-learning."],"supporting_citations":[{"why":"Defines the optimal control problem (1) and proves the algebraic equation (3) whose solution $p$ is the optimal cost vector.","marker":"Ohlin et al., 2024b"},{"why":"Proves that the problem class is equivalent to Stochastic Shortest Path and supplies Assumptions 1 and 2.","marker":"Ohlin et al., 2024a"},{"why":"Provides the Q-learning algorithm for SSP used as the numerical baseline.","marker":"Yu and Bertsekas, 2013"},{"why":"Introduces the data-driven online bounded-disturbance viewpoint that motivates the correlation-matrix update and condition (19).","marker":"Rantzer, 2024"},{"why":"Gives the comparison principle used in Lemma 2 to turn Bellman inequalities into bounds on the value vector.","marker":"Li and Rantzer, 2024"},{"why":"Defines the Q-function whose Bellman form is the basis of the data-driven algebraic equation (12).","marker":"Bradtke et al., 1994"}],"fun_headline_variants":["Adaptive controller stabilizes positive systems, bounds cumulative cost","Excitation-robustness tradeoff quantified for adaptive SSP learning","Data-driven stability guarantee for adaptive positive-system control","Stable adaptive control via data-driven algebraic equation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument needs $\\hat{\\alpha}=1-(1-\\rho\\beta)^{-1}\\rho\\beta$ to be positive, which is equivalent to $\\rho\\beta<1/2$, because the proof divides by $\\hat{\\alpha}$; Theorem 1 only states $\\rho\\beta<1$, so for mismatch levels between half and one the lower bound on $p(t)$ is not actually established.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive controller stabilizes positive systems, bounds cumulative cost","Excitation-robustness tradeoff quantified for adaptive SSP learning","Data-driven stability guarantee for adaptive positive-system control","Stable adaptive control via data-driven algebraic equation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3643,"prompt_tokens":966,"completion_tokens":2677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2612}},"tokens_in":582,"tokens_out":2677,"duration_ms":19957,"temperature":1.0,"reasoning_tokens":2612,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:55:52.368453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the constants at a pair with $\\rho\\beta\\in(1/2,1)$, for example $\\beta=1.5$ and $\\rho=0.4$: then $\\hat{\\alpha}=1-0.6/0.4=-0.5$, so the claimed lower bound $\\hat{\\alpha}p\\leq p(t)$ is vacuous and the proof's division by $\\hat{\\alpha}$ is illegitimate. A concrete check would simulate a plant in $\\mathcal{M}_\\beta$ with a disturbance sequence satisfying (19) at that $\\rho$ and test whether $p(t)$ still satisfies a non-vacuous two-sided bound; if it does, the theorem's range can be repaired, and if not, the stated range must be restricted.","supporting_citations":[{"cited_title":"On boundedness of q-learning iterates for stochastic shortest path problems","cited_arxiv_id":null,"evidence_quote":"Provides the Q-learning algorithm for SSP used as the numerical baseline."},{"cited_title":"Adaptive linear quadratic control using policy iteration","cited_arxiv_id":null,"evidence_quote":"Defines the Q-function whose Bellman form is the basis of the data-driven algebraic equation (12)."}],"review_version":1}