Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Control Synthesis with Reinforcement Learning: A Modeling Perspective

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL controllers trained on an inaccurate model fail in physical experiments.

desk verdict A genuine hardware case study that shows an LTI-trained RL controller failing while a nonlinear-model-trained controller works, but the comparison rests on single unseeded runs and an unvalidated 'accurate' model, so the general claim outruns the evidence. read the letter →

arxiv 2510.25063 v3 pith:ESLFPDUO submitted 2025-10-29 eess.SY cs.SY

Nikki Xu , Hien Tran This is my paper
classification eess.SYcs.SY MSC 93B3593E3593D15
keywords reinforcementlearningsim-to-realgapmodelmismatchinvertedpendulumsensitivityanalysisregionofattractionpolicygradientcontrolsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the quality of the simulation model, not the learning algorithm, is what determines whether a reinforcement-learning controller can be deployed on physical hardware. Using a cart-pole testbed, the authors train identical policy networks with the same algorithm in two environments: a detailed nonlinear model derived from first principles and a simplified linear time-invariant approximation. The controller trained on the detailed model balances the real pendulum and recovers from external taps, while the controller trained on the linear model fails or oscillates violently in repeated lab trials. Local sensitivity analysis shows that neglected rotational inertia and hinge damping are the main culprits, and an empirical region-of-attraction estimate quantifies the gap.

What carries the argument

The argument rests on comparing two training environments for the same policy-gradient algorithm: the 'lab model,' a fourth-order nonlinear system from Lagrangian mechanics including actuator and damping terms, and the 'linear model,' an LTI linearisation that neglects rotational inertia, hinge damping, and nonlinear coupling. A two-hidden-layer ReLU neural network policy is trained with REINFORCE in each environment. The authors then use complex-step local sensitivity analysis to rank parameter sensitivities and a Monte Carlo region-of-attraction estimation to compare closed-loop robustness. The central object is the pair of environment models: the differences in their parameter sensitiviti

What would settle it

Run the same training and deployment procedure with a simulator whose parameters are deliberately mis-specified by a few percent relative to the true hardware; if the lab-model-trained controller still fails on hardware, the paper's claim that an accurate model is necessary would be weakened, whereas if a coarse model with domain randomisation also succeeds, the claim that a high-fidelity model is required would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that a controller synthesised by reinforcement learning in simulation inherits the fidelity of the training environment: an 'accurate' first-principles model yields a policy that transfers directly to hardware without fine-tuning, whereas an 'inaccurate' simplified model yields a policy that performs almost identically in simulation but becomes brittle in the laboratory. The paper demonstrates this on a single inverted pendulum on a cart, where all five lab experiments using the linear-model-trained controller ended in failure or severe oscillation, while the lab-model-trained controller balanced the pendulum and rejected manual disturbances. Sensitivity analysis identif

Load-bearing premise

The comparison assumes that the 'lab model' parameters in Table 2 correctly describe the physical hardware, and that no other unmodeled dynamics (e.g., delays, sensor noise, motor saturation) are significant; if these parameters are wrong, the attribution of failure to the simplified model's neglected terms is not fully established.

Editorial extensions

If this is right

  • For RL-based control to be deployed safely, the training simulator must include high-sensitivity physical effects (here, motor shaft inertia and hinge damping) even if they are small in magnitude.
  • Local sensitivity analysis with complex-step differentiation is cheap enough to be added to standard RL training loops and can flag which neglected parameters will break a learned policy.
  • An empirically estimated region of attraction can serve as a sanity check for sim-to-real transfer before physical experiments.
  • Simplified models remain useful for analysis, but they may need to be augmented with the sensitive terms rather than discarded in favour of black-box domain randomisation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The finding suggests that model selection for RL training should be driven by parameter sensitivity analysis rather than by textbook familiarity; the same methodology could identify critical neglected physics in other underactuated systems such as quadrotors or manipulators.
  • The paper implicitly argues against the default use of out-of-the-box simulation benchmarks, implying that controller comparisons in RL are only meaningful when the training environment is calibrated to the target hardware.
  • A testable extension would replace the hand-picked simplified model with a data-driven black-box model and check whether a policy trained on it transfers; if it does, the conclusion is about model fidelity rather than physics-based modeling per se.
  • The observed asymmetry in disturbance rejection (better for counterclockwise taps) suggests that enforcing approximate symmetry in the policy architecture could further improve deployment reliability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how model fidelity in the simulation environment affects the deployability of reinforcement-learning-based controllers for a single inverted pendulum on a cart. The authors train the same REINFORCE algorithm with a two-hidden-layer ReLU policy on two environments: a detailed first-principles 'lab model' (Eq. 3) and a simplified LTI model obtained by neglecting damping, rotational inertia, and nonlinear terms (Eq. 5). They report that both controllers stabilize the respective simulation for 20 seconds, but lab experiments show the LTI-trained controller fails or oscillates severely in all five trials, whereas the lab-model-trained controller balances and rejects perturbations (Sections 3.2.1–3.2.2). The paper then presents a complex-step local sensitivity analysis of model parameters and a Monte Carlo region-of-attraction estimation on 80,979 initial conditions, concluding that the high-fidelity training model gives a larger basin of attraction and more robust closed-loop behavior. The central claim is that designing RL controllers in an inaccurate simulation environment is unsuitable for physical deployment, while training with an accurate model yields robust controllers.

Significance. If the central claim is fully substantiated, the paper would provide a useful, concrete demonstration of the sim-to-real gap in a classic benchmark, with a practical diagnostic (local sensitivity analysis) and an empirical robustness assessment. The work has clear strengths: it uses a physical lab setup, derives the nonlinear model from first principles, and reports a substantial Monte Carlo region-of-attraction study (80,979 samples). The qualitative failure contrast—five of five LTI-trained controllers failing or oscillating versus successful balancing with the lab-model-trained controller—is striking and worth reporting. However, as described below, the comparison rests on single training runs with separately tuned hyperparameters and the 'accurate' model is asserted rather than validated against the hardware. These issues do not invalidate the qualitative observation but they do prevent the paper from supporting the strong, general claim in the abstract at its current level of evidence.

major comments (4)
  1. [Section 3.1 and 3.2] The central comparison is confounded by training variability. Section 3.1 states that the NAdam hyperparameters were 'tuned separately' for the two experiments, but no random seeds, episode counts, learning curves, or numbers of training runs are reported. REINFORCE is a high-variance policy gradient method, so the single LTI-trained policy deployed in Section 3.2.1 may be an unrepresentative draw. The claim that 'a poor model results in a controller that performs well in simulation but fails in physical experiments' requires that the only causally relevant difference between the two deployed controllers is training-model fidelity. The authors should retrain each policy over multiple seeds, report the distribution of simulation performance, and deploy policies matched on simulation performance. Without this, the hardware outcome cannot be uniquely attributed to model accuracy.
  2. [Section 2.1, Table 2, and Section 3.2] The 'accurate model' is not independently validated. The lab model in Eq. (3) is derived from first principles, and the parameters in Table 2 are presented as given, but there is no system-identification residual analysis, no comparison of predicted versus measured open-loop trajectories, and no parameter-fitting procedure. Because the same hardware is used both to motivate the lab model and as the test set, the claim that the lab model is 'accurate' is circular unless model error is quantified. The authors should report open-loop prediction errors or parameter-identification results that justify calling the lab model high-fidelity relative to the LTI model.
  3. [Section 4.2, Figure 18] The region-of-attraction comparison does not provide a clean quantitative measure of robustness. The reported success rates (26,583/80,979, 16,318/80,979, and 46,178/80,979) are computed from a non-uniform sampling protocol: the paper states that initial conditions were sampled in neighborhoods around 'previously validated initial conditions' and that radii were progressively increased. This yields a density-biased sample, and the 'hole' near the origin is dismissed rather than analyzed. The qualitative ordering of the three systems may be correct, but the percentages should not be interpreted as volumes or probabilities under a well-defined initial-condition distribution. A uniform or clearly stratified sampling design, or reporting the success rate as a function of distance from the origin, would strengthen the claim.
  4. [Section 4.1 and Section 5] The sensitivity analysis is used to 'justify' the discrepancy, but the connection to hardware failure is qualitative. Figures 14–16 show that the lab-model-trained controller has lower sensitivity to J_m and K_m in the controlled lab-model system, and that the LTI model neglects J_m and B_p. However, no statement is made about how large a parameter perturbation the LTI-trained controller can tolerate, or how the measured sensitivity magnitudes relate to actual hardware parameter uncertainty. The sensitivity results are suggestive and useful, but they do not by themselves establish that J_m or B_p mismatch caused the observed failures. A small quantitative experiment, such as perturbing J_m or B_p in the lab model and measuring the success rate of the LTI-trained policy, would make the causal story much stronger.
minor comments (5)
  1. [Abstract and Section 1] Minor English issues: 'Controllers designed using an accurate model is robust' should be 'are robust'; the phrase 'a poor model results in a controller that performs well in simulation but fails in physical experiments' is a strong claim that should be softened to 'in this case' unless the multi-seed evidence is added.
  2. [Section 3.1, Eq. (12)] Equation (12) writes V_m/10 on the left-hand side but the following explicit line gives V_m = 10(...). Either drop the '/10' in Eq. (12) or clarify that the output is normalized.
  3. [Section 4.2] The reference 'Fig. ' is empty in the sentence 'The denser result are shown in Fig. ', and the figure number should be filled in.
  4. [Section 4.2] The success criterion for the ROA study uses a final-second angle tolerance of 0.05 radians, while the training reward uses 0.2 radians. The mismatch is not discussed; please clarify why the stricter criterion was chosen and whether the results are sensitive to it.
  5. [Appendix, Table 2] The list of parameters is missing units for some entries (e.g., B_p and B_c are labeled with units in the table but the text sometimes omits them) and the table caption does not note the source of the parameter values (manufacturer data, direct measurement, or estimation). This is related to the major comment on model validation.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the main result is an empirical sim-to-real comparison, not a fitted or definitionally forced prediction.

full rationale

The paper's central claim is supported by a direct hardware comparison between two REINFORCE-trained controllers, one trained on the detailed lab model and one on its LTI simplification. The lab model is derived from a Lagrangian first-principles derivation with parameters given in Table 2; it is not fitted to the hardware data that is later used as the test set, and no fitted parameter is renamed as a prediction. The LTI model is an explicit simplification of the same dynamics with stated assumptions. Training and deployment are empirical evaluations, and the sensitivity analysis and region-of-attraction estimates are post-hoc analyses of already-trained policies, not quantities whose outputs are enforced by construction. The self-citations that appear, e.g., [KT16] for the lab-specific model equations, are not load-bearing in a circular sense because the equations are reproduced in the paper and the empirical outcome is independent of the citation. Residual concerns about separate hyperparameter tuning, unreported random seeds, and unvalidated parameter accuracy are experimental validity issues rather than circularity, and do not justify raising the circularity score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The paper depends on the accuracy of the nonlinear lab model and on the REINFORCE training runs; the key unstated quantities are the training hyperparameters and the model-validation evidence. The sensitivity analysis is an explanatory diagnostic, not a new fitted quantity.

free parameters (2)
  • Neural network hyperparameters and initialization seeds = not reported
    The two policies were trained with NAdam with separately tuned learning rates (Section 3.1), but the values and seeds are not given. The comparison could depend on the quality of the individual training runs.
  • ROA sampling bias/sample counts = 5000 initial; then 80,979 total in neighborhoods around previously-validated points
    The ROA estimates use Monte Carlo sampling; the authors state the initial 5000-sample estimate is not statistically significant, and the denser samples are centered on previously validated initial conditions, biasing the final percentages (Section 4.2).
assumptions (3)
  • domain assumption The lab model (Eq. 3) with the parameter values in Table 2 accurately represents the physical cart-pole hardware.
    The entire comparison treats the detailed model as the accurate environment; the parameters are taken from prior literature/lab setup without reporting system identification or model-validation residuals (Section 2.1, Table 2).
  • standard math REINFORCE trajectories in the manuscript's notation follow the standard policy gradient derivation (Eqs. 6-7).
    The policy-gradient objective and the trajectory probability in Eq. 6 assume the Markov property and a stochastic policy; this is standard and stated in Section 2.3.
  • domain assumption Complex-step sensitivity accurately approximates sensitivities of the closed-loop ReLU-controlled system.
    Complex-step differentiation assumes analyticity of the function being differentiated; the closed-loop system includes the non-smooth ReLU controller, so the standard complex-step justification does not strictly apply. The paper cites Banks et al. for non-smooth problems, but no proof or error analysis is given for the ReLU case (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Control Synthesis with Reinforcement Learning: A Modeling Perspective." pith.science (2026). https://pith.science/paper/ESLFPDUO

@misc{pith2026251025063,
  author       = {Pith},
  title        = {Pith review of: Control Synthesis with Reinforcement Learning: A Modeling Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESLFPDUO}},
  note         = {Machine review of arXiv:2510.25063}
}
read the original abstract

Controllers designed with reinforcement learning can be sensitive to model mismatch. We demonstrate that designing such controllers in a virtual simulation environment with an inaccurate model is not suitable for deployment in a physical setup. Controllers designed using an accurate model is robust against disturbance and small mismatch between the physical setup and the mathematical model derived from first principles; while a poor model results in a controller that performs well in simulation but fails in physical experiments. Sensitivity analysis is used to justify these discrepancies and an empirical region of attraction estimation help us visualize their robustness.

Figures

Figures reproduced from arXiv: 2510.25063 by the authors.

Figure 1
Figure 1. Sketch of Pendulum on Cart where Fc is motor driving force, Bc is the damping coefficient of the horizontal movement of the cart, and Bp is the damping coefficient of the rotation of the pendulum. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Architecture of the Neural Network Parameterizing a Normal Distribution for [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Control Learned with Different Systems The lab implementation of this controller is a deterministic one, that is, we use the weights and biases of all the neurons except for the final output of standard deviation, 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Control Learned with LTI Systems Simulated with Difference Systems [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 7
Figure 7. Figure 7: Furthermore, the cart was very shaky in both experiments, and the motor was [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 5
Figure 5. Figure 5: Controller Learned with LTI Model Generally Fails in Lab Experiment [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Control Learned with LTI Systems Tested in the Lab Quickly Fails [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Control Learned with LTI Systems Tested in the Lab Balances the Pendulum for [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: run #5: Oscillation Amplified and Eventually Fails to Balance the Pendulum [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Control Learned with Lab Model Test in Lab [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Control learned with lab model tested in lab. Top plot of each subfigure is the [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Sensitivity of State Variables with respect to the Viscous Damping Coefficient [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Sensitivity of State Variables with respect to the Rotational Moment of Inertia [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Sensitivity of State Variables with respect to the Equivalent Viscous Damping on [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Sensitivity of State Variables with respect to the Rotational Moment of Inertia [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Sensitivity of State Variables with respect to the Radius of Motor Torque Con [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Full Trajectory of Sensitivity of State Variables with respect to the Radius of [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Region of Attraction of Different Control Systems [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Region of Attraction: Sampled in Neighborhoods Around Previously Found Valid [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Sensitivity of State Variables with respect to the Mass of Cart Parameter, [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: Sensitivity of State Variables with respect to the Mass of Pendulum Parameter, [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]
Figure 21
Figure 21. Figure 21: Sensitivity of State Variables with respect to the Length of Pendulum Parameter, [PITH_FULL_IMAGE:figures/full_fig_p033_21.png]
Figure 22
Figure 22. Figure 22: Sensitivity of State Variables with respect to the Gravity Parameter, [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]
Figure 23
Figure 23. Figure 23: Sensitivity of State Variables with respect to the Radius of Motor Pinion Param [PITH_FULL_IMAGE:figures/full_fig_p035_23.png]
Figure 24
Figure 24. Figure 24: Sensitivity of State Variables with respect to the Motor Armature Resistance [PITH_FULL_IMAGE:figures/full_fig_p036_24.png]
Figure 25
Figure 25. Figure 25: Sensitivity of State Variables with respect to the Gear Ratio Parameter, [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]
Figure 26
Figure 26. Figure 26: Sensitivity of State Variables with respect to the Electromotive Force Constant [PITH_FULL_IMAGE:figures/full_fig_p038_26.png]
Figure 27
Figure 27. Figure 27: Sensitivity of State Variables with respect to the Motor Torque Constant Param [PITH_FULL_IMAGE:figures/full_fig_p039_27.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System

    cs.RO 2026-06 unverdicted novelty 3.0 of 10

    Zero-shot sim-to-real transfer of independently trained RL policies for cart-pole swing-up and stabilization is achieved via sensitivity-guided domain randomization, linear curriculum learning, and first-order action ...

Reference graph

Works this paper leans on

50 extracted references · 14 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada

    Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control Barrier Functions : Theory and Applications . In 2019 18th European Control Conference ( ECC ) , pages 3420--3431, June 2019

    2019

  2. [2]

    Anderson, Samuel Pfrommer, and Somayeh Sojoudi

    Brendon G. Anderson, Samuel Pfrommer, and Somayeh Sojoudi. Tight Certified Robustness via Min - Max Representations of ReLU Neural Networks , October 2023. arXiv:2310.04916 [cs, math]

    arXiv 2023

  3. [3]

    Examining the simulation-to-reality gap of a wheel loader digging in deformable terrain

    Koji Aoshima and Martin Servin. Examining the simulation-to-reality gap of a wheel loader digging in deformable terrain. Multibody System Dynamics , 64(1):121--148, May 2025

    2025

  4. [4]

    Wanniarachchi

    Buddhika Abeysekera and Indika L. Wanniarachchi. Modelling and Implementation of PID Control for Balancing of an Inverted Pendulum . 2018

    2018

  5. [5]

    Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković

    Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learning : Grids , Groups , Graphs , Geodesics , and Gauges , May 2021. arXiv:2104.13478 [cs]

    arXiv 2021

  6. [6]

    The complex-step method for sensitivity analysis of non-smooth problems arising in biology

    H T Banks, Kidist Bekele-Maxwell, Lorena Bociu, Marcella Noorman, and Kristen Tillman. The complex-step method for sensitivity analysis of non-smooth problems arising in biology. Eurasian Journal of Mathematical and Computer Applications , 3:15--68, 2015

    2015

  7. [7]

    Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P

    Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe Learning in Robotics : From Learning - Based Control to Safe Reinforcement Learning . Annual Review of Control, Robotics, and Autonomous Systems , 5(Volume 5, 2022):411--444, May 2022. Publisher: Annual Reviews

    2022

  8. [8]

    Benchmarking Virtual Reinforcement Learning Algorithms to Balance a Real Inverted Pendulum

    Dylan Bates and Hien Tran. Benchmarking Virtual Reinforcement Learning Algorithms to Balance a Real Inverted Pendulum . In Kohei Arai, editor, Intelligent Systems and Applications , pages 285--303, Cham, 2022. Springer International Publishing

    2022

Show all 50 references
  1. [9]

    Schoellig, and Andreas Krause

    Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe Model -based Reinforcement Learning with Stability Guarantees , November 2017. arXiv:1705.08551 [cs, stat]

    2017 arXiv

  2. [10]

    AUGCAL : Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images , July 2024

    Prithvijit Chattopadhyay, Bharat Goyal, Boglarka Ecsedi, Viraj Prabhu, and Judy Hoffman. AUGCAL : Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images , July 2024. arXiv:2312.06106 [cs]

    2024 arXiv

  3. [11]

    Auto- Tuned Sim -to- Real Transfer , May 2021

    Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto- Tuned Sim -to- Real Transfer , May 2021. arXiv:2104.07662 [cs]

    2021 arXiv

  4. [12]

    Kévin Garanger, Julie Kraus, and Julian J. Rimoli. Symmetry-enforcing neural networks with applications to constitutive modeling. Extreme Mechanics Letters , 71:102188, September 2024

    2024

  5. [13]

    Bayesian Reinforcement Learning : A Survey

    Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian Reinforcement Learning : A Survey . Foundations and Trends® in Machine Learning , 8(5-6):359--483, November 2015. Publisher: Now Publishers, Inc

    2015

  6. [14]

    Garud N. Iyengar. Robust Dynamic Programming . Mathematics of Operations Research , 30(2):257--280, May 2005. Publisher: INFORMS

    2005

  7. [15]

    A Comparison of LQR and MPC Control Algorithms of an Inverted Pendulum

    Andrzej Jezierski, Jakub Mozaryn, and Damian Suski. A Comparison of LQR and MPC Control Algorithms of an Inverted Pendulum . In Wojciech Mitkowski, Janusz Kacprzyk, Krzysztof Oprzedkiewicz, and Pawel Skruch, editors, Trends in Advanced Intelligent Control , Optimization and Au...

    2017

  8. [16]

    Symmetric Replay Training : Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization , July 2024

    Hyeonah Kim, Minsu Kim, Sungsoo Ahn, and Jinkyoo Park. Symmetric Replay Training : Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization , July 2024. arXiv:2306.01276 [cs] version: 4

    2024 arXiv

  9. [17]

    Kennedy and Hien T

    Emese A. Kennedy and Hien T. Tran. Real- Time Stabilization of a Single Inverted Pendulum Using a Power Series Based Controller . In Gi-Chul Yang, Sio-Iong Ao, Xu Huang, and Oscar Castillo, editors, Transactions on Engineering Technologies , pages 1--14, Singapore, 2016. Springer

    2016

  10. [18]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019. arXiv:1509.02971 [cs, stat]

    2019 arXiv

  11. [19]

    J. N. Lyness and C. B. Moler. Numerical Differentiation of Analytic Functions . SIAM Journal on Numerical Analysis , 4(2):202--210, 1967. Publisher: Society for Industrial and Applied Mathematics

    1967

  12. [20]

    Accelerated Primal - Dual Policy Optimization for Safe Reinforcement Learning , February 2018

    Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated Primal - Dual Policy Optimization for Safe Reinforcement Learning , February 2018. arXiv:1802.06480 [cs, stat]

    2018 arXiv

  13. [21]

    Lewis, Draguna L

    Frank L. Lewis, Draguna L. Vrabie, and Vassilis L. Syrmos. Optimal Control . Wiley, 3 edition, January 2012

    2012

  14. [22]

    Robust Reinforcement Learning

    Jun Morimoto and Kenji Doya. Robust Reinforcement Learning . Neural Computation , 17(2):335--359, February 2005. Conference Name: Neural Computation

    2005

  15. [23]

    Application of Kalman Filtering and PID Control for Direct Inverted Pendelum Control

    Jose Luis Corona Miranda. Application of Kalman Filtering and PID Control for Direct Inverted Pendelum Control . Publisher: California State University, Chico

  16. [24]

    Joaquim R. R. A. Martins, Ilan Kroo, and Juan Alonso. An automated method for sensitivity analysis using complex variables. In 38th Aerospace Sciences Meeting and Exhibit , Reno,NV,U.S.A., January 2000. American Institute of Aeronautics and Astronautics

    2000

  17. [25]

    Learn to Swing Up and Balance a Real Pole Based on Raw Visual Input Data

    Jan Mattner, Sascha Lange, and Martin Riedmiller. Learn to Swing Up and Balance a Real Pole Based on Raw Visual Input Data . In Tingwen Huang, Zhigang Zeng, Chuandong Li, and Chi Sing Leung, editors, Neural Information Processing , pages 126--133, Berlin, Heidelberg, 2012. Springer

    2012

  18. [26]

    Doerr, Edgar Granados, Aravind Sivaramakrishnan, Abdeslam Boularias, and Kostas E

    Isidoros Marougkas, Dhruv Metha Ramesh, Joe H. Doerr, Edgar Granados, Aravind Sivaramakrishnan, Abdeslam Boularias, and Kostas E. Bekris. Integrating Model -based Control and RL for Sim2Real Transfer of Tight Insertion Policies , May 2025. arXiv:2505.11858 [cs]

    2025 arXiv

  19. [27]

    Robot Learning From Randomized Simulations : A Review

    Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot Learning From Randomized Simulations : A Review . Frontiers in Robotics and AI , 9, April 2022. Publisher: Frontiers

    2022

  20. [28]

    Robustness in Markov Decision Problems with Uncertain Transition Matrices

    Arnab Nilim and Laurent El Ghaoui. Robustness in Markov Decision Problems with Uncertain Transition Matrices

  21. [29]

    Braatz, and Rolf Findeisen

    Hoang Hai Nguyen, Tim Zieger, Richard D. Braatz, and Rolf Findeisen. Robust Control Theory Based Stability Certificates for Neural Network Approximated Nonlinear Model Predictive Control . IFAC-PapersOnLine , 54(6):347--352, 2021

    2021

  22. [30]

    Design and implementation of LQR controller for inverted pendulum by use of REX control system

    Stepan Ozana, Martin Pies, Zdenek Slanina, and Radovan Hajovsky. Design and implementation of LQR controller for inverted pendulum by use of REX control system. In 2012 12th International Conference on Control , Automation and Systems , pages 343--347, October 2012

    2012

  23. [31]

    Sim-to- Real Transfer of Robotic Control with Dynamics Randomization

    Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to- Real Transfer of Robotic Control with Dynamics Randomization . In 2018 IEEE International Conference on Robotics and Automation ( ICRA ) , pages 3803--3810, May 2018. arXiv:1710.06537 [cs]

    2018 arXiv

  24. [32]

    Robust Adversarial Reinforcement Learning

    Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Reinforcement Learning . In Proceedings of the 34th International Conference on Machine Learning , pages 2817--2826. PMLR, July 2017. ISSN: 2640-3498

    2017

  25. [33]

    Polydoros and Lazaros Nalpantidis

    Athanasios S. Polydoros and Lazaros Nalpantidis. Survey of Model - Based Reinforcement Learning : Applications on Robotics . Journal of Intelligent & Robotic Systems , 86(2):153--173, May 2017

    2017

  26. [34]

    Chance constrained policy optimization for process control and optimization

    Panagiotis Petsagkourakis, Ilya Orson Sandoval, Eric Bradford, Federico Galvanin, Dongda Zhang, and Ehecatl Antonio del Rio-Chanona. Chance constrained policy optimization for process control and optimization. Journal of Process Control , 111:35--45, March 2022

    2022

  27. [35]

    Richards, Felix Berkenkamp, and Andreas Krause

    Spencer M. Richards, Felix Berkenkamp, and Andreas Krause. The Lyapunov Neural Network : Adaptive Stability Certification for Safe Learning of Dynamical Systems , October 2018. arXiv:1808.00924 [cs]

    2018 arXiv

  28. [36]

    A Tour of Reinforcement Learning : The View from Continuous Control

    Benjamin Recht. A Tour of Reinforcement Learning : The View from Continuous Control . Annual Review of Control, Robotics, and Autonomous Systems , 2(Volume 2, 2019):253--279, May 2019. Publisher: Annual Reviews

    2019

  29. [37]

    Riedmiller

    M. Riedmiller. Neural reinforcement learning to swing-up and balance a real pole. In 2005 IEEE International Conference on Systems , Man and Cybernetics , volume 4, pages 3191--3196 Vol. 4, October 2005. ISSN: 1062-922X

    2005

  30. [38]

    Todorov, and Sham M Kakade

    Aravind Rajeswaran, Kendall Lowrey, Emanuel V. Todorov, and Sham M Kakade. Towards Generalization and Simplicity in Continuous Control . In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proc...

    2017

  31. [39]

    BayesSim : adaptive domain randomization via probabilistic inference for robotics simulators, June 2019

    Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. BayesSim : adaptive domain randomization via probabilistic inference for robotics simulators, June 2019. arXiv:1906.01728 [cs]

    2019 arXiv

  32. [40]

    Sutton and Andrew Barto

    Richard S. Sutton and Andrew Barto. Reinforcement learning: an introduction . Adaptive computation and machine learning. The MIT Press, Cambridge, Massachusetts London, England, second edition edition, 2020

    2020

  33. [41]

    Sutton, A.G

    R.S. Sutton, A.G. Barto, and R.J. Williams. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems Magazine , 12(2):19--22, April 1992. Conference Name: IEEE Control Systems Magazine

    1992

  34. [42]

    Hong, and Sehoon Ha

    Nitish Sontakke, Hosik Chae, Sangjoon Lee, Tianle Huang, Dennis W. Hong, and Sehoon Ha. Residual Physics Learning and System Identification for Sim -to-real Transfer of Policies on Buoyancy Assisted Legged Robots , March 2023. arXiv:2303.09597 [cs]

    2023 arXiv

  35. [43]

    Using Complex Variables to Estimate Derivatives of Real Functions

    William Squire and George Trapp. Using Complex Variables to Estimate Derivatives of Real Functions . SIAM Review , 40(1):110--112, January 1998. Publisher: Society for Industrial and Applied Mathematics

    1998

  36. [44]

    How to certify machine learning based safety-critical systems? A systematic literature review

    Florian Tambon, Gabriel Laberge, Le An, Amin Nikanjam, Paulina Stevia Nouwou Mindom, Yann Pequignot, Foutse Khomh, Giulio Antoniol, Ettore Merlo, and François Laviolette. How to certify machine learning based safety-critical systems? A systematic literature review. Automated S...

    2022

  37. [45]

    Mark Towers, Jordan K Terry, Ariel Kwiatkowski, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium...

    2023

  38. [46]

    Robust Backstepping Sliding Mode Control Design for a Class of Underactuated Electro – Mechanical Nonlinear Systems

    Safeer Ullah, Qudrat Khan, Adeel Mehmood, and Aamer Iqbal Bhatti. Robust Backstepping Sliding Mode Control Design for a Class of Underactuated Electro – Mechanical Nonlinear Systems . Journal of Electrical Engineering & Technology , 15(4):1821--1828, July 2020

    2020

  39. [47]

    Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning , November 2022

    Tyler Westenbroek, Fernando Castaneda, Ayush Agrawal, Shankar Sastry, and Koushil Sreenath. Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning , November 2022. arXiv:2208.06721 [cs]

    2022 arXiv

  40. [48]

    Distributionally Robust Markov Decision Processes

    Huan Xu and Shie Mannor. Distributionally Robust Markov Decision Processes . Mathematics of Operations Research , 37(2):288--300, 2012. Publisher: INFORMS

    2012

  41. [49]

    Sim-to- Real Transfer in Deep Reinforcement Learning for Robotics : a Survey

    Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to- Real Transfer in Deep Reinforcement Learning for Robotics : a Survey . In 2020 IEEE Symposium Series on Computational Intelligence ( SSCI ) , pages 737--744, December 2020

    2020

  42. [50]

    Symmetry in Neural Network Parameter Spaces , October 2025

    Bo Zhao, Robin Walters, and Rose Yu. Symmetry in Neural Network Parameter Spaces , October 2025. arXiv:2506.13018 [cs]

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.