Pith. sign in

REVIEW 4 major objections 6 minor 56 references

Data-Driven LQR with Finite-Time Experiments via Extremum-Seeking Policy Iteration

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Exponential LQR convergence from short experiments only

desk verdict A solid averaging-theory proof for a genuinely new finite-experiment LQR scheme, but the convergence guarantee only covers the exact noiseless oracle, not real experiments. read the letter →

arxiv 2412.02758 v3 pith:TVF4OAJC submitted 2024-12-03 math.OC cs.SYeess.SY

Guido Carnevale , Nicola Mimmo , Giuseppe Notarstefano This is my paper · ORCID
classification math.OCcs.SYeess.SY MSC 93B5293C5549N1090C56
keywords data-drivenLQRextremum-seekingcontrolpolicyiterationfinite-horizoncostaveragingtheorydiscrete-timenonlinearsystemsderivative-freeoptimizationmodel-freegradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes EXP-LQR, an iterative algorithm that solves Linear Quadratic Regulator problems using only finitely many short experiments on the system, each returning a truncated approximation of the infinite-horizon cost. The algorithm needs no knowledge of the system matrices or the cost matrices. The paper proves that EXP-LQR exponentially converges to an arbitrarily small neighborhood of the optimal state-feedback gain, while keeping every policy used in the experiments stabilizing. This matters because it gives a theoretically certified, derivative-free route to LQR optimality in settings where only finite-time data are available, such as simulators or real plants.

What carries the argument

The engine is an extremum-seeking update combined with a deterministic sinusoidal dither matrix $D_k$ and a scalar low-pass filter $z_k$. At each iteration the current gain $K_k$ is perturbed to $K_k+\delta D_k$, the truncated cost $J_T(K_k+\delta D_k)$ is measured from experiments, and the filter tracks that cost while the gain is moved along the dither-normalized difference signal. The analysis uses a newly stated discrete-time averaging theorem (Theorem 1) to compare this time-varying system with its period-averaged version, whose dynamics are a perturbed policy gradient. A Lyapunov function $V_\lambda(z,K)=\frac{1}{2\lambda}\|z\|^2+J(K)-J(K^\star)$ combines the filter error with the cost gap, and gradient dominance of the LQR cost gives the exponential decay; the combination yields semi-global practical exponential stability.

What would settle it

Run EXP-LQR on a known scalar or small-state LQR with exact finite-horizon cost oracle and compare $\|K_k-K^\star\|$ against the claimed exponential envelope $a_0(1-\gamma a)^k+r$ across many choices of $(\gamma,\delta,T)$; if the error never enters the $r$-ball along the predicted envelope for any admissible parameter choice, Theorem 2 is false. A complementary test is to add small adversarial noise to $J_T(K)$, a case the proof excludes, and observe whether convergence degrades; that would delineate the theorem's limit rather than falsify it.

Watch

Extended reading notes

Core claim

The central claim is Theorem 2: for any desired accuracy radius $r>0$ and any stabilizing initial gain $K_0$, there exist sufficiently small step size $\gamma$, sufficiently small dither amplitude $\delta$, and sufficiently large experiment horizon $T$ such that the EXP-LQR updates keep $K_k+\delta D_k$ stabilizing at every step and satisfy $\|K_k-K^\star\|\le a_0(1-\gamma a)^k+r$ for all $k$. In other words, the extremum-seeking policy iteration converges exponentially fast into any prescribed ball around the optimal LQR gain $K^\star$, using only finite-horizon truncated cost measurements. The convergence statement is semi-global in the set of stabilizing gains and practical in the sense that the final accuracy can be made arbitrarily good, and the proof treats the algorithm as a discrete-time time-varying nonlinear system analyzed via averaging and a Lyapunov argument.

Load-bearing premise

The proof needs the oracle to return the exact truncated cost $J_T(K)$ as the mean of $n$ noise-free finite-time experiments starting from the coordinate basis vectors; if measurements carry noise or the initial states cannot be set exactly, the averaging argument as written no longer holds.

Editorial extensions

If this is right

  • A practitioner can run EXP-LQR on a simulator or real plant by prescribing $T$-step experiments, and Theorem 2 certifies exponential approach to the optimal gain once $\gamma$, $\delta$, and $T$ are chosen appropriately.
  • No identification of $(A,B)$ or $(Q,R)$ is needed; only the scalar value of the truncated LQR cost is used, so the method applies when cost matrices are known only through sensor data or user feedback.
  • Because the dither is deterministic and the averaged system is a policy-gradient update, the analysis connects extremum seeking directly with gradient-descent theory for LQR.
  • Every policy used during learning remains stabilizing, so the algorithm never drives the closed-loop system unstable while collecting data.
  • The standalone discrete-time averaging theorem can be reused for other time-varying algorithms whose averaged dynamics are stable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is a stochastic version: if the truncated-cost oracle is noisy, the averaging argument as written does not apply, but a stochastic-approximation variant with noise-robust proofs would be a natural next step.
  • The paper proves existence of suitable parameters but does not give explicit quantitative rates for how $\gamma$, $\delta$, and $T$ should be chosen; a concrete tuning rule would make the method easier to deploy.
  • Because only $n$ experiments are needed per iteration for an $n$-dimensional state, the per-iteration experiment count is independent of the input dimension $m$, which could make the method attractive for systems with many actuators.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes EXP-LQR, a data-driven algorithm for discrete-time LQR that replaces model knowledge with evaluations of a finite-horizon truncated cost. At each iteration the current gain is perturbed by deterministic sinusoidal dither matrices, the truncated cost of the perturbed gain is measured in n experiments initialized on the canonical basis, and a low-pass-filtered extremum-seeking update is applied. The main theoretical result, Theorem 2, states that under Assumptions 4-6 and for sufficiently small step size, small dither amplitude, and sufficiently long experiment horizon, the gain iterates remain stabilizing and converge exponentially to an arbitrarily small neighborhood of the optimal LQR gain. The proof uses a generic discrete-time averaging result (Theorem 1), a gradient approximation lemma (Lemma 2), and a Lyapunov analysis of the averaged system (Lemma 3). A numerical study on a doubly fed induction motor model compares EXP-LQR with a model-free policy gradient method.

Significance. If correct, EXP-LQR is a useful addition to the data-driven LQR literature: it requires only finite-time cost evaluations rather than knowledge of (A,B) or (Q,R), uses deterministic dithering and filtering, and provides non-asymptotic exponential convergence guarantees. The generic discrete-time averaging theorem (Theorem 1) is also potentially reusable. The paper is honest about the need for a stabilizing initial gain and about the difficulty of tuning parameters. However, the advertised 'real experiments' claim is currently supported only for an exact, noiseless oracle, and one key lemma is imported from a related paper. These points do not invalidate the exact-oracle theorem, but they do limit the practical reach of the paper as written.

major comments (4)
  1. [§IV, Eqs. (9)-(11), Theorem 2] The convergence guarantee is stated and proved only for the exact truncated-cost oracle in (9)-(10). If the measured cost is corrupted as J_T^obs(K_k+δD_k)=J_T(K_k+δD_k)+ε_k, the update (11b) becomes K_{k+1}=K_k - γ(2/δ)(J_T(K_k+δD_k)-z_k)D_k - γ(2/δ)ε_kD_k, so the error term is amplified by 1/δ. None of the bounds in Section V (Lemma 1, Eq. (44), Eq. (64)) contains ε_k or an admissible noise amplitude, and the theorem's conclusion (13) does not cover such perturbations. The abstract and introduction advertise 'real experiments'; as written, Theorem 2 covers only noiseless virtual or exactly repeatable experiments. Please either restrict the claims accordingly or provide a noise-robustness analysis with explicit noise bounds that preserve (13).
  2. [Appendix C, Lemma 2] Lemma 2 is not proved in this paper. Appendix C states that [56, Lemma 1] provides the same result and that the steps can be repeated after replacing global C^3 regularity by compactness of S. Since Lemma 2 is the key gradient-approximation bound that defines p_{δ,T} in (18b) and underpins Lemma 3, a load-bearing result is imported. The paper should either give a full proof, including the Taylor expansion and the uniform bound on the remainder, or explicitly state Theorem 2 as conditional on [56, Lemma 1] with the precise statement of that lemma included.
  3. [§V-C, Eq. (23)] After applying Theorem 1, the proof writes Vλ(z_k - J(K_k), K_k) ≤ c1 and Vλ(z_k - J(K_k), K_k) ≤ (1-γa)^k Vλ(z0 - J(K0), K0) + ρ. However, Lemma 3 and Assumption 3 are established for Vλ(z_k - J^δ_AV(K_k), K_k), where J^δ_AV is defined in (18a); the averaged filter's fixed point is J^δ_AV, not J. The displayed inequalities therefore do not follow from the preceding application of Theorem 1, and the derivation of (13b) must be redone with the correct Lyapunov function. This appears repairable by a change of notation plus a uniform approximation argument, but as written it is a load-bearing gap.
  4. [Appendix A, Eq. (37)] The proof of Theorem 1 uses the fact that χ_k^AV ∈ Ω_{c0} for all k≥0, citing Assumption 3 in Eq. (4). Assumption 3 actually gives V(χ_k^AV) ≤ (1-γa)^k V(χ0) + ρAV, hence at most c0+ρAV, not c0. Consequently the step V(χ_k)≤c0+ϵ≤c1 requires an additional smallness condition on ρAV and ϵ (e.g., c0+ρAV+ϵ<c1), which is not stated. Since Theorem 1 is the mechanism for transferring averaged convergence to the original system, this gap should be fixed in the rewritten proof.
minor comments (6)
  1. [§VI, Fig. 4] The caption lists MFPG with γ=10^-9, while the text says MFPG is run with γ=10^-8 and otherwise with the same tuning as EXP-LQR; the correspondence between curves and step sizes should be clarified.
  2. [Appendix D, around Eq. (56)] The displayed bound uses 'min' where a bound on the norm requires a maximum; as written, the inequality is not a valid upper bound.
  3. [§V-B, Eq. (18a)] The definition of J^δ_AV(K) uses τ from k+1 to k+kprd, but the quantity is independent of k because of periodicity; making the index range independent of k would improve clarity.
  4. [§V, general notation] The averaged system state is denoted K^k_AV while K also denotes the set of stabilizing gains; this is a minor notational clash that makes some formulas harder to parse.
  5. [§VII] The conclusions refer to an 'inductance motor'; the paper studies an induction motor.
  6. [Algorithm 1] In the experiment phase, 'Test x_{t+1}=A x_t + B u_t' should be 'Set' or 'Run', since no test is being performed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EXP-LQR's convergence proof derives from truncation-error, averaging, and Lyapunov arguments, not from its own assumptions.

full rationale

The derivation chain is self-contained. Theorem 2 is proved by (i) Lemma 1, which bounds the finite-horizon truncation error J - J_T via convergence of the truncated cost series on compact subsets of the stabilizing set K; (ii) Lemma 2, which expresses the averaged zeroth-order gradient estimator as G(K) + delta^2 e(K); and (iii) Lemma 3, which establishes semi-global practical exponential stability of the averaged system using a Lyapunov function V_lambda, gradient dominance of the LQR cost [24, Lemma 3.12], and standard bounds; finally Theorem 1 transfers the averaged-system property to the original time-varying algorithm. None of these steps defines the optimal gain K* in terms of the measured costs or fits a parameter to the convergence claim. The only overlapping-author citation is [56, Lemma 1], used to justify the derivative-free gradient approximation in Lemma 2; that lemma is a parameter-free statement about C^3 functions under sinusoidal dithers and does not contain the LQR convergence result, so it is independent support rather than a circular premise. The practical gap concerning noisy finite-time experiments is a robustness/correctness limitation, not a circularity: the theorem is stated for the exact oracle in (9)-(10) and does not assume its conclusion. The numerical section tunes parameters empirically but presents the behavior as a simulation matching Theorem 2, not as a fitted prediction of the theory. Overall, no circular step is present; any concern is about the gap between the exact-oracle theorem and real noisy experiments, which belongs to correctness risk rather than self-referential derivation.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The theorem is not fully self-contained: it imports several published results, including one from the authors' own group ([56]) for the gradient approximation. The finite-time oracle exactness and the stabilizing initial gain are domain assumptions rather than derived facts. No physical entities are invented; the dither matrix is an algorithmic device, not a new physical quantity. The free parameters are algorithm hyperparameters, not fitted model constants.

free parameters (4)
  • Step size gamma = 10^-7 in text; 10^-8 in Fig. 4 caption
    Chosen empirically per Remark 2. The theorem only requires gamma to be sufficiently small and gives no constructive bound.
  • Perturbation amplitude delta = 10^-2
    Chosen empirically to be small enough so that K + delta D remains stabilizing and the gradient approximation error is controlled.
  • Experiment horizon T = 20 samples
    Must be large enough for the truncated cost to approximate the infinite-horizon cost; tuned empirically rather than by the explicit bound in Lemma 1.
  • Dither frequencies and phases = kp,prd = 19 * 2^((1-p)/2) for odd p, same for even p with phase pi/2
    Hand-designed to satisfy Assumption 6. Not fitted to data, but the proof depends on these orthonormality conditions.
assumptions (8)
  • domain assumption (A,B) controllable and Q,R symmetric positive definite (Assumption 4).
    Standard LQR well-posedness condition used throughout Theorem 2.
  • domain assumption (A,B) and (Q,R) are unknown (Assumption 5).
    This is the data-driven setting the algorithm is designed for.
  • domain assumption Initial gain K0 is stabilizing, K0 in K.
    Theorem 2 assumes a stabilizing initial policy. The paper notes data-driven ways to obtain one but does not provide the construction.
  • ad hoc to paper Dither signals satisfy orthonormality conditions (12a)-(12c) with common period kprd (Assumption 6).
    This algorithm-specific design requirement is needed to make the derivative-free gradient approximation error O(delta^2).
  • standard math J is smooth and gradient-dominated with compact sublevel sets; K is open and connected (external results [24], [49]).
    Used in Lemma 3 and the proof of Theorem 2 for Lyapunov decrease and invariance arguments.
  • standard math Quadratic growth bound psi ||K - K*||^2 <= J(K) - J(K*) from [51, Lemma 3.8].
    Converts the Lyapunov decrease into the distance-to-optimum bound in (13b).
  • standard math Derivative-free gradient approximation lemma for C^3 functions ([56, Lemma 1]).
    Imported to prove Lemma 2. The authors state the proof repeats [56] under compactness rather than reproducing it.
  • domain assumption The experimenter can set initial states to e_i and the measured cost is exactly VT(K,e_i) with no noise (Eqs. 9-10).
    Enables exact finite-horizon cost evaluations. This is load-bearing for the deterministic averaging proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data-Driven LQR with Finite-Time Experiments via Extremum-Seeking Policy Iteration." pith.science (2026). https://pith.science/paper/TVF4OAJC

@misc{pith2026241202758,
  author       = {Pith},
  title        = {Pith review of: Data-Driven LQR with Finite-Time Experiments via Extremum-Seeking Policy Iteration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TVF4OAJC}},
  note         = {Machine review of arXiv:2412.02758}
}
read the original abstract

In this paper, we address Linear Quadratic Regulator (LQR) problems through a novel iterative algorithm named EXtremum-seeking Policy iteration LQR (EXP-LQR). The peculiarity of EXP-LQR is that it only needs access to a truncated approximation of the infinite-horizon cost associated to a given policy. Hence, EXP-LQR does not need the direct knowledge of neither the system and cost matrices. In particular, at each iteration, EXP-LQR refines the maintained policy using a truncated LQR cost retrieved by performing finite-time virtual or real experiments in which a perturbed version of the current policy is employed. Such a perturbation is done according to an extremum-seeking mechanism and makes the overall algorithm a time-varying nonlinear system. By using a Lyapunov-based approach exploiting averaging theory, we show that EXP-LQR exponentially converges to an arbitrarily small neighborhood of the optimal gain matrix. We corroborate the theoretical results with numerical simulations involving the control of an induction motor.

Figures

Figures reproduced from arXiv: 2412.02758 by the authors.

Figure 1
Figure 1. Using this information, EXP-LQR iteratively improves [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Block diagram representation of Algorithm 1. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Block diagram representation of the averaged sys [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Evolution of the maximum (in absolute value) closed [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Comparison between EXP-LQR and MFPG [23] in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 43 canonical work pages

  1. [1]

    From model-based control to data-driven control: Survey, classification and perspective,

    Z.-S. Hou and Z. Wang, “From model-based control to data-driven control: Survey, classification and perspective,” Information Sciences , vol. 235, pp. 3–35, 2013

    2013

  2. [2]

    On an iterative technique for Riccati equation com- putations,

    D. Kleinman, “On an iterative technique for Riccati equation com- putations,” IEEE Transactions on Automatic Control , vol. 13, no. 1, pp. 114–115, 1968

    work page 1968

  3. [3]

    Online optimal tracking control of continuous-time linear systems with unknown dynamics by using adaptive dynamic programming,

    C. Qin, H. Zhang, and Y . Luo, “Online optimal tracking control of continuous-time linear systems with unknown dynamics by using adaptive dynamic programming,” International Journal of Control , vol. 87, no. 5, pp. 1000–1009, 2014

    work page 2014

  4. [4]

    Optimal output-feedback control of unknown continuous-time linear systems using off-policy reinforcement learning,

    H. Modares, F. L. Lewis, and Z.-P. Jiang, “Optimal output-feedback control of unknown continuous-time linear systems using off-policy reinforcement learning,” IEEE Transactions on Cybernetics , vol. 46, no. 11, pp. 2401–2410, 2016

    work page 2016

  5. [5]

    Data-driven finite-horizon optimal control for linear time-varying discrete-time systems,

    B. Pang, T. Bian, and Z.-P. Jiang, “Data-driven finite-horizon optimal control for linear time-varying discrete-time systems,” in 2018 IEEE Conference on Decision and Control (CDC) , pp. 861–866, IEEE, 2018

    work page 2018

  6. [6]

    Finite-time analysis of approximate policy iteration for the linear quadratic regulator,

    K. Krauth, S. Tu, and B. Recht, “Finite-time analysis of approximate policy iteration for the linear quadratic regulator,” Advances in Neural Information Processing Systems , vol. 32, 2019

    work page 2019

  7. [7]

    Robust policy iteration for continuous- time linear quadratic regulation,

    B. Pang, T. Bian, and Z.-P. Jiang, “Robust policy iteration for continuous- time linear quadratic regulation,” IEEE Transactions on Automatic Control, vol. 67, no. 1, pp. 504–511, 2021

    work page 2021

  8. [8]

    Efficient off-policy Q- learning for data-based discrete-time LQR problems,

    V . G. Lopez, M. Alsalti, and M. A. M ¨uller, “Efficient off-policy Q- learning for data-based discrete-time LQR problems,” IEEE Transactions on Automatic Control , vol. 68, no. 5, pp. 2922–2933, 2023

    work page 2023

Show all 56 references
  1. [9]

    Safely learning to control the constrained linear quadratic regulator,

    S. Dean, S. Tu, N. Matni, and B. Recht, “Safely learning to control the constrained linear quadratic regulator,” in IEEE American Control Conference (ACC), pp. 5582–5588, 2019

    2019

  2. [10]

    Certainty equivalence is efficient for linear quadratic control,

    H. Mania, S. Tu, and B. Recht, “Certainty equivalence is efficient for linear quadratic control,” Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  3. [11]

    Learning robust LQ-controllers using application oriented exploration,

    M. Ferizbegovic, J. Umenberger, H. Hjalmarsson, and T. B. Sch ¨on, “Learning robust LQ-controllers using application oriented exploration,” IEEE Control Systems Letters , vol. 4, no. 1, pp. 19–24, 2019

    2019

  4. [12]

    Formulas for data-driven control: Stabilization, optimality, and robustness,

    C. De Persis and P. Tesi, “Formulas for data-driven control: Stabilization, optimality, and robustness,” IEEE Transactions on Automatic Control , vol. 65, no. 3, pp. 909–924, 2019

    2019

  5. [13]

    Data informativity: a new perspective on data-driven analysis and control,

    H. J. Van Waarde, J. Eising, H. L. Trentelman, and M. K. Camlibel, “Data informativity: a new perspective on data-driven analysis and control,” IEEE Transactions on Automatic Control, vol. 65, no. 11, pp. 4753–4768, 2020

    2020

  6. [14]

    Data-driven linear quadratic regulation via semidefinite programming,

    M. Rotulo, C. De Persis, and P. Tesi, “Data-driven linear quadratic regulation via semidefinite programming,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 3995–4000, 2020

    2020

  7. [15]

    Online learning of data-driven controllers for unknown switched linear systems,

    M. Rotulo, C. De Persis, and P. Tesi, “Online learning of data-driven controllers for unknown switched linear systems,” Automatica, vol. 145, p. 110519, 2022

    2022

  8. [16]

    From noisy data to feedback controllers: Nonconservative design via a matrix S-lemma,

    H. J. van Waarde, M. K. Camlibel, and M. Mesbahi, “From noisy data to feedback controllers: Nonconservative design via a matrix S-lemma,” IEEE Transactions on Automatic Control , vol. 67, no. 1, pp. 162–175, 2020

    2020

  9. [17]

    Low-complexity learning of linear quadratic regulators from noisy data,

    C. De Persis and P. Tesi, “Low-complexity learning of linear quadratic regulators from noisy data,” Automatica, vol. 128, p. 109548, 2021

    2021

  10. [18]

    On the certainty-equivalence approach to direct data-driven LQR design,

    F. D ¨orfler, P. Tesi, and C. De Persis, “On the certainty-equivalence approach to direct data-driven LQR design,” IEEE Transactions on Automatic Control, vol. 68, no. 12, pp. 7989–7996, 2023

    2023

  11. [19]

    Robust data- driven state-feedback design,

    J. Berberich, A. Koch, C. W. Scherer, and F. Allg ¨ower, “Robust data- driven state-feedback design,” in 2020 American Control Conference (ACC), pp. 1532–1538, IEEE, 2020

    2020

  12. [20]

    CoRe: Control-oriented regularization for system identification,

    S. Formentin and A. Chiuso, “CoRe: Control-oriented regularization for system identification,” in 2018 IEEE Conference on Decision and Control (CDC), pp. 2253–2258, IEEE, 2018

    2018

  13. [21]

    Structured exploration in the finite horizon linear quadratic dual control problem,

    A. Iannelli, M. Khosravi, and R. S. Smith, “Structured exploration in the finite horizon linear quadratic dual control problem,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 959–964, 2020

    2020

  14. [22]

    Bridging direct and indirect data-driven control formulations via regularizations and relaxations,

    F. D¨orfler, J. Coulson, and I. Markovsky, “Bridging direct and indirect data-driven control formulations via regularizations and relaxations,” IEEE Transactions on Automatic Control , vol. 68, no. 2, pp. 883–897, 2022

    2022

  15. [23]

    Global convergence of policy gradient methods for the linear quadratic regulator,

    M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” in International conference on machine learning , pp. 1467–1476, PMLR, 2018. 12

    2018

  16. [24]

    LQR through the lens of first order methods: Discrete-time case,

    J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi, “LQR through the lens of first order methods: Discrete-time case,” arXiv preprint arXiv:1907.08921, 2019

    1907 arXiv

  17. [25]

    Policy optimization for H2 linear control with H∞ robustness guarantee: Implicit regularization and global convergence,

    K. Zhang, B. Hu, and T. Basar, “Policy optimization for H2 linear control with H∞ robustness guarantee: Implicit regularization and global convergence,” in Learning for Dynamics and Control , pp. 179–190, PMLR, 2020

    2020

  18. [26]

    Global convergence of policy gradient methods to (almost) locally optimal policies,

    K. Zhang, A. Koppel, H. Zhu, and T. Basar, “Global convergence of policy gradient methods to (almost) locally optimal policies,” SIAM Journal on Control and Optimization , vol. 58, no. 6, pp. 3586–3612, 2020

    2020

  19. [27]

    Revisiting LQR control from the perspective of receding-horizon policy gradient,

    X. Zhang and T. Ba s ¸ar, “Revisiting LQR control from the perspective of receding-horizon policy gradient,” IEEE Control Systems Letters , vol. 7, pp. 1664–1669, 2023

    2023

  20. [28]

    Toward a theoretical foundation of policy optimization for learning control policies,

    B. Hu, K. Zhang, N. Li, M. Mesbahi, M. Fazel, and T. Ba s ¸ar, “Toward a theoretical foundation of policy optimization for learning control policies,” Annual Review of Control, Robotics, and Autonomous Systems , vol. 6, pp. 123–158, 2023

    2023

  21. [29]

    Adaptive optimal control for continuous-time linear systems based on policy iteration,

    D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptive optimal control for continuous-time linear systems based on policy iteration,” Automatica, vol. 45, no. 2, pp. 477–484, 2009

    2009

  22. [30]

    Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics,

    Y . Jiang and Z.-P. Jiang, “Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics,” Automatica, vol. 48, no. 10, pp. 2699–2704, 2012

    2012

  23. [31]

    Value iteration for continuous-time linear time-invariant systems,

    C. Possieri and M. Sassano, “Value iteration for continuous-time linear time-invariant systems,” IEEE Transactions on Automatic Control, vol. 68, no. 5, pp. 3070–3077, 2022

    2022

  24. [32]

    On-policy data-driven linear quadratic regulator via model reference adaptive reinforcement learning,

    M. Borghesi, A. Bosso, and G. Notarstefano, “On-policy data-driven linear quadratic regulator via model reference adaptive reinforcement learning,” in IEEE 62nd Conference on Decision and Control (CDC) , pp. 32–37, IEEE, 2022

    2022

  25. [33]

    MR-ARL: Model reference adaptive reinforcement learning for robustly stable on-policy data-driven LQR,

    M. Borghesi, A. Bosso, and G. Notarstefano, “MR-ARL: Model reference adaptive reinforcement learning for robustly stable on-policy data-driven LQR,” arXiv preprint arXiv:2402.14483 , 2024

    2024 arXiv

  26. [34]

    On-policy data-driven linear quadratic regulator via combined policy iteration and recursive least squares,

    L. Sforni, G. Carnevale, I. Notarnicola, and G. Notarstefano, “On-policy data-driven linear quadratic regulator via combined policy iteration and recursive least squares,” in IEEE 62nd Conference on Decision and Control (CDC), pp. 5047–5052, 2023

    2023

  27. [35]

    Stability- certified on-policy data-driven LQR via recursive learning and policy gradient,

    L. Sforni, G. Carnevale, I. Notarnicola, and G. Notarstefano, “Stability- certified on-policy data-driven LQR via recursive learning and policy gradient,” arXiv preprint arXiv:2403.05367 , 2024

    2024 arXiv

  28. [36]

    The role of identification in data-driven policy iteration: A system theoretic study,

    B. Song and A. Iannelli, “The role of identification in data-driven policy iteration: A system theoretic study,” International Journal of Robust and Nonlinear Control, 2024

    2024

  29. [37]

    100 years of extremum seeking: A survey,

    A. Scheinker, “100 years of extremum seeking: A survey,” Automatica, vol. 161, p. 111481, 2024

    2024

  30. [38]

    Adaptive extremal control,

    B. Wittenmark and A. Urquhart, “Adaptive extremal control,” in Proceedings of 1995 34th IEEE Conference on Decision and Control , vol. 2, pp. 1639–1644, IEEE, 1995

    1995

  31. [39]

    Solving smooth and nonsmooth multivariable extremum seeking problems by the methods of nonlinear programming,

    A. R. Teel and D. Popovic, “Solving smooth and nonsmooth multivariable extremum seeking problems by the methods of nonlinear programming,” in Proceedings of the 2001 American Control Conference.(Cat. No. 01CH37148), vol. 3, pp. 2394–2399, IEEE, 2001

    2001

  32. [40]

    Ariyur and M

    K. Ariyur and M. Krstic, Real-Time Optimization by Extremum-Seeking Control. Wiley-Interscience, Wiley, 2003

    2003

  33. [41]

    Stability of extremum seeking feedback for general nonlinear dynamic systems,

    M. Krsti ´c and H.-H. Wang, “Stability of extremum seeking feedback for general nonlinear dynamic systems,” Automatica, vol. 36, no. 4, pp. 595–601, 2000

    2000

  34. [42]

    On non-local stability properties of extremum seeking control,

    Y . Tan, D. Neˇsi´c, and I. Mareels, “On non-local stability properties of extremum seeking control,” Automatica, vol. 42, no. 6, pp. 889–903, 2006

    2006

  35. [43]

    Finite-horizon LQ control for unknown discrete-time linear systems via extremum seeking,

    P. Frihauf, M. Krstic, and T. Ba s ¸ar, “Finite-horizon LQ control for unknown discrete-time linear systems via extremum seeking,” European Journal of Control , vol. 19, no. 5, pp. 399–407, 2013

    2013

  36. [44]

    Nonlinear systems,

    H. K. Khalil, “Nonlinear systems,” Upper Saddle River , 2002

    2002

  37. [45]

    J. A. Sanders, F. Verhulst, and J. Murdock, Averaging methods in nonlinear dynamical systems , vol. 59. Springer, 2007

    2007

  38. [46]

    Averaging analysis for discrete time and sampled data adaptive systems,

    E.-W. Bai, L.-C. Fu, and S. S. Sastry, “Averaging analysis for discrete time and sampled data adaptive systems,” IEEE Transactions on Circuits and Systems, vol. 35, no. 2, pp. 137–148, 1988

    1988

  39. [47]

    Extremum-seeking policy iteration for data-driven LQR,

    G. Carnevale, N. Mimmo, and G. Notarstefano, “Extremum-seeking policy iteration for data-driven LQR,” in IEEE 63rd Conference on Decision and Control (CDC), IEEE , 2024

    2024

  40. [48]

    B. D. Anderson and J. B. Moore, Optimal control: linear quadratic methods. Courier Corporation, 2007

    2007

  41. [49]

    On topological properties of the set of stabilizing feedback gains,

    J. Bu, A. Mesbahi, and M. Mesbahi, “On topological properties of the set of stabilizing feedback gains,” IEEE Transactions on Automatic Control , vol. 66, no. 2, pp. 730–744, 2020

    2020

  42. [50]

    Personalized optimization with user’s feedback,

    A. Simonetto, E. Dall’Anese, J. Monteil, and A. Bernstein, “Personalized optimization with user’s feedback,” Automatica, vol. 131, p. 109767, 2021

    2021

  43. [51]

    LQR via first order flows,

    J. Bu, A. Mesbahi, and M. Mesbahi, “LQR via first order flows,” in 2020 American Control Conference (ACC) , pp. 4683–4688, IEEE, 2020

    2020

  44. [52]

    Leonhard, Control of electrical drives

    W. Leonhard, Control of electrical drives . Springer Science & Business Media, 2001

    2001

  45. [53]

    Sines and cosines of angles in arithmetic progression,

    M. P. K. and, “Sines and cosines of angles in arithmetic progression,” Mathematics Magazine, vol. 82, no. 5, pp. 371–372, 2009

    2009

  46. [54]

    On the discrete analogy of Gronwall lemma,

    J. Popenda, “On the discrete analogy of Gronwall lemma,” Demonstratio Mathematica, vol. 16, no. 1, pp. 11–26, 1983

    1983

  47. [55]

    Discrete Gronwall lemma and applications,

    J. M. Holte, “Discrete Gronwall lemma and applications,” in MAA-NCS meeting at the University of North Dakota , vol. 24, pp. 1–7, 2009

    2009

  48. [56]

    Extremum seeking tracking for derivative-free distributed optimization,

    N. Mimmo, G. Carnevale, A. Testa, and G. Notarstefano, “Extremum seeking tracking for derivative-free distributed optimization,” IEEE Transactions on Control of Network Systems , 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.