Pith. sign in

REVIEW 4 major objections 4 minor 35 references

Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that by adapting reward and penalty gains online based on how close a robot is to violating physical constraints, a robot can learn to walk from scratch in the real world without falling and without tedious offline gain tu

desk verdict ROGER's adaptive gain rule is genuinely new and the real-robot results are credible, but the Appendix B 'guarantee' does not hold up even on the paper's own assumptions. read the letter →

arxiv 2510.10759 v2 pith:3XO5U3KE submitted 2025-10-12 cs.RO

Arthicha Srisuchinnawong , Poramate Manoonpong This is my paper · ORCID
classification cs.RO
keywords constrainedreinforcementlearningrewardshapingrobotlocomotiononlinegainadaptationconstraintsatisfactionquadrupedreal-worldRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ROGER, a rule that adjusts the weighting between a robot's positive reward and its constraint penalties during learning, based on how close the measured behavior is to user-set safety thresholds. The paper tries to establish that this online adjustment removes the need for offline reward-gain tuning while keeping constraint violations near zero throughout training, something fixed-weight and existing adaptive-weight methods do not reliably achieve. If the claim is right, it matters because safe-in-training learning is the main barrier to teaching heavy robots to walk directly in the real world, where a fall can damage the machine or its surroundings.

What carries the argument

The central object is the gain-allocation rule ROGER. It computes an urgency signal from the squared ratios of estimated constraint penalties to their thresholds; the primary reward gain is set to one minus that signal, and the signal is distributed among the penalty gains according to each penalty's share of the total urgency. This ratio-based reallocation is what makes the same reward function act performance-oriented in safe states and safety-oriented near constraints, with the only user-set hyperparameters being intuitive threshold values such as a maximum allowable body tilt.

What would settle it

Train the same 60-kg quadruped with ROGER at four times the nominal learning rate and count how often body roll or pitch exceeds the 10-degree threshold; the paper's own appendix reports that failures appear under this condition. A cleaner test: inject a controlled perturbation that forces the total reward to decrease sharply mid-episode and check whether ROGER's gains reallocate before the constraint is crossed; if violations occur, the near-zero-violation claim holds only under the gentle-dynamics condition stated in the paper.

Watch

Extended reading notes

Core claim

The central claim is that a single gain-allocation rule, ROGER, can replace manual reward tuning and still satisfy constraints during learning. At every timestep the rule lowers the primary reward gain and raises the penalty gains in proportion to the squared ratio of each estimated penalty to its threshold; near a threshold the primary reward is almost ignored and the offending constraint dominates, while far from thresholds the primary reward dominates and performance is prioritized. In a simulated 60-kg quadruped, this yields one constraint violation in 50,000 timesteps and a final forward speed of 0.6 m/s, 50% higher than careful fixed-gain and reward-switching baselines; on the physical

Load-bearing premise

The safety guarantee rests on the assumption that the total reward never decreases during learning and that both the robot's dynamics and the learning process are gentle enough for that to hold; when exploration or a high learning rate makes the reward drop abruptly, ROGER can violate constraints, which the paper itself demonstrates for excessive learning rates.

Editorial extensions

If this is right

  • Real-world learning from a standstill becomes practical for heavy robots: exploration near a safety boundary is countered immediately by gain reallocation, so the robot need not fall to learn.
  • Reward tuning reduces to setting constraint tolerances (e.g., 10 degrees of tilt) rather than grid-searching dozens of weighting constants, cutting both human time and the risk of picking unsafe weights.
  • Constraint satisfaction does not have to come at the cost of task reward in safe regions: the final primary reward matches the unconstrained objective while keeping near-zero violations.
  • ROGER is an add-on to the reward function, so it can be combined with different control architectures and learning rules rather than replacing them.
  • The same rule generalizes to more dynamic simulated tasks, where it improves distance and reduces torque and orientation deviation relative to default weights, even though the theoretical safety guarantee is weaker there.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of ROGER suggests a general design principle for safe RL: make reward weights explicit functions of constraint proximity rather than tuned constants. A natural next test would be applying the same ratio-allocation rule to manipulation tasks with force or position limits.
  • Because the rule reacts to statistical estimates of penalties, its safety margin depends on how well those estimates track the true state. An untested extension would be to feed the rule with deliberately noisy or biased sensor estimates to map the boundary of the safety margin.
  • The paper leaves global-position and other exteroceptive constraints as future work; the same logic could plausibly handle them by treating accumulated position error as a penalty channel with a threshold, but the accumulating nature of such errors is exactly where the current proof's assumptions would be stressed.
  • If the total-reward-nondecreasing assumption is violated by non-smooth or high-learning-rate updates, ROGER's guarantee weakens; a stress-test variant that deliberately perturbs learning with occasional large decreases would show whether the practical margin covers what the theory does not.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ROGER, an online reward-gain adaptation mechanism for constrained reinforcement learning in robot locomotion. ROGER adjusts the primary-reward and penalty gains at each timestep based on the ratio of estimated penalties to user-specified constraint thresholds, shrinking the primary gain and growing penalty gains near constraint boundaries. The authors claim that ROGER ensures constraint satisfaction during training, improves primary reward over fixed-gain and adaptive baselines, and reduces reward-tuning burden. They support this with simulated and physical experiments on a 60-kg Unitree B1 quadruped, simulated MuJoCo locomotion benchmarks (hopper, Ant, Half Cheetah, Walker2D), and a real-world learning demonstration with no falls in under an hour. A Lyapunov-style analysis in Appendix B is offered as theoretical backing.

Significance. If the empirical claims hold, ROGER would be a practically useful and simple add-on for real-world continual locomotion learning, where constraint violations can be destructive and reward-shaping is labor-intensive. The strengths of the paper are its extensive empirical evaluation: multiple seeds/repetitions, statistical tests, hyperparameter sensitivity analyses, a real-world hardware demonstration, and a clear comparison against fixed-gain, CBF, PDO, CRPO, and OL-AUX baselines. The authors also provide code and videos. However, the paper's theoretical contribution—the claimed guarantee of constraint satisfaction and monotone primary-reward increase in Appendix B—is not internally sound. The derivation contains a load-bearing logical error, and the main premise (dR_t/dt ≥ 0) is neither proven nor consistent with the authors' own exploration experiments. Therefore the central advertised claim of 'ensuring constraint satisfaction throughout learning' is not backed by the theory, and the paper currently overstates what is rigorously established.

major comments (4)
  1. [Appendix B, Eq. (S2)–(S4)] The learning analysis is invalid as written. Starting from dR_t ≥ 0 and the simplified total reward R_t = R0 − R0 R1² − R1³, the authors obtain the differential inequality dR0 ≥ [R1(2R0 + 3R1)/(1 − R1²)] dR1. They then solve the associated equality as if it were an ODE, integrate, and identify the constant of integration with R0t0. This step is unjustified: an inequality cannot be integrated as an ODE, and the constant cannot be set to R0t0 without proof. The sketch's claim is actually false: for R0 decreasing from 1.0 to 0.4 while R1 decreases from 0.8 to 0.2, R_t = R0 − R0R1² − R1³ increases from −0.152 to 0.376, so dR_t/dt > 0, yet the primary reward R0 decreases. Thus Eq. (S4), and the derived guarantee that ∫dR0 ≥ R0t0, does not follow. This undermines the theoretical basis for the 'expected primary reward guaranteed to increase' and 'partial stability' claims. The authors should ei
  2. [Appendix B, premise dR_t/dt ≥ 0] The Lyapunov and learning analyses both rely on the assertion 'Since the learning rule maximizes R_t, it follows that Ṙ_t ≥ 0.' This is not generally true for AGOL or PPO with exploration, and the paper's own Figure S8a shows that excessive learning rates lead to constraint violations, i.e., R_t need not be monotone. The premise is thus unproven and empirically violated in the paper's own experiments. At minimum, the theorem statements in Appendix B must be explicitly conditional on this assumption, and the paper should not state without qualification that ROGER 'ensures constraint satisfaction' or 'guarantees an increase in primary reward' (Section IV, Abstract).
  3. [Section III.A.1, violation probabilities] There are arithmetic errors in the reported violation probabilities. The text states that ROGER exhibited 'one in 50,000 timesteps' and then equates this to 'an estimated violation probability of around 2×10⁻¹²%.' 1/50,000 = 2×10⁻⁵, which is 2×10⁻³%, not 2×10⁻¹²%. Similarly, the fixed-gain penalty is credited with 'three violations in 25,000 timesteps' but an estimated violation probability of 2×10⁻¹². These numbers differ by many orders of magnitude. This is not a presentation typo only: the quantitative comparison of violation rates is a central empirical claim. The authors should correct these values and restate the corresponding statistical comparisons.
  4. [Section II / Conclusion, 'ensuring constraint satisfaction'] The paper's abstract and Section II state that ROGER 'ensures constraint satisfaction throughout learning' and that the adaptation strategy is 'proven to be partially stable.' Given the issues above, the theoretical support for these claims is not present. The empirical evidence does show very low violation counts in the quadruped experiments, but the theory does not establish a guarantee. I recommend that the authors rephrase these claims as empirical findings—for example, 'ROGER achieved near-zero violations in these experiments'—and clearly separate what is proven from what is observed.
minor comments (4)
  1. [Section I.B (intro)] Two near-identical paragraphs begin with 'Recent techniques proposed in early 2025 build on these approaches.' The duplicate should be removed.
  2. [Figures 1 and 3 captions] Figure 3's caption is identical to the paragraph in Section II, and the full caption is repeated in the figure. Consider shortening the caption to its descriptive part only.
  3. [Equations (2)–(5)] The notation is generally clear, but the paper should state explicitly that Δ_t and r_λit are recomputed only at update intervals, not necessarily every timestep, and how the statistical estimate R̃_it is computed over the stored trajectory window.
  4. [Section III.A.1, 'videos'] The YouTube links have extra spaces in the text (e.g., 'Cqu7vLT Piw?si=...'). This is a typesetting issue but should be fixed for the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains and violations are measured against independent baselines, and ROGER's gain rule is computed online from penalties rather than fitted to the target results.

full rationale

The paper's central quantitative claims—near-zero constraint violation, 50% higher primary reward than fixed-gain/CRPO, and real-world walking in under an hour—are empirical comparisons against independently implemented baselines (fixed-gain penalty, quadratic/logarithmic CBF, PDO, CRPO, OL-AUX, and default Gymnasium rewards). ROGER's adaptive gains (Eqs. 2–5) are deterministic functions of the current estimated penalties and user-chosen thresholds; they are not fitted to the reported final speeds or violation counts, so the 'predictions' are not equivalent to fit inputs. Appendix B does contain a genuine mathematical weakness: the step from the differential inequality (S2) to the integrated bound (S4) treats dR1 as an independent differential and incorrectly identifies the integration constant with R0t0, and the dRt≥0 premise is itself unproven for AGOL/PPO. But this is a proof-soundness/correctness issue, not circularity: the claimed guarantee is not assumed as an input to the derivation, and the experimental results do not depend on the proof. The only author-overlap citation, SME-AGOL [21], supplies an experimental backbone that is separately published and benchmarked; it is not invoked to define ROGER's predictions or to forbid alternatives. The stated limitations (zero-penalty optimality, gentle system/learning dynamics) are explicit, and the paper runs a hopper experiment under violated assumptions, providing external falsifiability. Thus no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method's central claim rests on hand-chosen thresholds tau_i and uncertainty multiplier k_sigma, plus assumptions of zero-penalty optimality, gentle dynamics, and a non-decreasing total reward during learning.

free parameters (2)
  • k_sigma (penalty uncertainty multiplier) = 3 (varied 2-4; chosen for 99.9% confidence)
    Used in R_tilde = mean - k_sigma std; hand-selected to widen the safety margin; affects how early gains switch.
  • Constraint thresholds tau_i = roll/pitch 0.2 rad (~10 deg), torque 1.0 Nm, orientation 10-45 deg, height 0.2 m
    Chosen by hand from physical limits and environment; define the safe region and directly trigger gain changes.
assumptions (4)
  • domain assumption Zero-penalty optimality: the optimal policy has all constraint penalties at zero (R_1 -> 0).
    Assumed in Appendix B for the stability/learning analysis; violated by the torque penalty in the hopper experiment.
  • domain assumption Gentle system and learning dynamics: R_0 and R_1 do not change abruptly during learning.
    Stated in Section IV and Appendix B as necessary for partial stability; limits applicability to sudden dynamics or high learning rates.
  • ad hoc to paper The learning rule maximizes total reward so that dR_t/dt >= 0 at each step.
    Invoked in Appendix B to derive the Lyapunov inequality; not proven for AGOL or PPO and not guaranteed during exploration.
  • domain assumption Penalty estimates R_tilde = mean - k_sigma std over recent episodes reliably predict imminent constraint violations.
    ROGER's gain update uses this summary statistic; assumes stationarity within the estimation window and approximately symmetric distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning." pith.science (2026). https://pith.science/paper/3XO5U3KE

@misc{pith2026251010759,
  author       = {Pith},
  title        = {Pith review of: Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XO5U3KE}},
  note         = {Machine review of arXiv:2510.10759}
}
read the original abstract

Existing robot locomotion learning techniques rely heavily on the offline selection of proper reward weighting gains and cannot guarantee constraint satisfaction (i.e., constraint violation) during training. Thus, this work aims to address both issues by proposing Reward-Oriented Gains via Embodied Regulation (ROGER), which adapts reward-weighting gains online based on penalties received throughout the embodied interaction process. The ratio between the positive reward (primary reward) and negative reward (penalty) gains is automatically reduced as the learning approaches the constraint thresholds to avoid violation. Conversely, the ratio is increased when learning is in safe states to prioritize performance. With a 60-kg quadruped robot, ROGER achieved near-zero constraint violation throughout multiple learning trials. It also achieved up to 50% more primary reward than the equivalent state-of-the-art techniques. In MuJoCo continuous locomotion benchmarks, including a single-leg hopper, ROGER exhibited comparable or up to 100% higher performance and 60% less torque usage and orientation deviation compared to those trained with the default reward function. Finally, real-world locomotion learning of a physical quadruped robot was achieved from scratch within one hour without any falls. Therefore, this work contributes to constraint-satisfying real-world continual robot locomotion learning and simplifies reward weighting gain tuning, potentially facilitating the development of physical robots and those that learn in the real world.

Figures

Figures reproduced from arXiv: 2510.10759 by the authors.

Figure 1
Figure 1. (a) Parameter trajectories from (white) RL, (black) constrained RL, and (brown) ROGER on a simulated reward [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of how embodied interaction between the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. This adaptation strategy is also proven to be partially [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: (a) Final primary reward values obtained from the last training episode and (b-c) roll and pitch angles recorded [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Evolution of the main reward term across 500 learn [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: (Left) Evolution of the primary weighting gain ( [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: (a) Snapshots capturing the locomotion learning of a physical robot after 50 and 250 episodes. (b, top) Forward speed [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Snapshots, hopping distance, joint torque usage, and [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 3 linked inside Pith

  1. [1]

    Embodied AI beyond embodied cognition and enactivism.Philosophies, 4(3):39, 2019

    Riccardo Manzotti. Embodied AI beyond embodied cognition and enactivism.Philosophies, 4(3):39, 2019

    2019

  2. [2]

    MIT press, 2006

    Rolf Pfeifer and Josh Bongard.How the body shapes the way we think: a new view of intelligence. MIT press, 2006

    2006

  3. [3]

    Reinforcement learning.Journal of Cognitive Neuroscience, 11(1):126– 134, 1999

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning.Journal of Cognitive Neuroscience, 11(1):126– 134, 1999

    1999

  4. [4]

    Anymal parkour: Learning agile navigation for quadrupedal robots.Science Robotics, 9(88):eadi7566, 2024

    David Hoeller, Nikita Rudin, Dhionis Sako, and Marco Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots.Science Robotics, 9(88):eadi7566, 2024

    2024

  5. [5]

    Rapid locomotion via reinforcement learning.The International Journal of Robotics Research, 43(4):572–587, 2024

    Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning.The International Journal of Robotics Research, 43(4):572–587, 2024

    2024

  6. [6]

    Not only rewards but also constraints: Applications on legged robot locomotion.IEEE Trans- actions on Robotics, 2024

    Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Ji, Moonkyu Jung, Donghoon Youm, and Jemin Hwangbo. Not only rewards but also constraints: Applications on legged robot locomotion.IEEE Trans- actions on Robotics, 2024

    2024

  7. [7]

    Demon- strating a walk in the park: Learning to walk in 20 min- utes with model-free reinforcement learning.Robotics: Science and Systems (RSS) Demo, 2(3):4, 2023

    Laura Smith, Ilya Kostrikov, and Sergey Levine. Demon- strating a walk in the park: Learning to walk in 20 min- utes with model-free reinforcement learning.Robotics: Science and Systems (RSS) Demo, 2(3):4, 2023

    2023

  8. [8]

    Explor- ing constrained reinforcement learning algorithms for quadrupedal locomotion

    Joonho Lee, Lukas Schroth, Victor Klemm, Marko Bjelonic, Alexander Reske, and Marco Hutter. Explor- ing constrained reinforcement learning algorithms for quadrupedal locomotion. In2024 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS 2024), 2024

    2024

Show all 35 references
  1. [9]

    A review of safe reinforcement learning: Methods, theories and applications.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Flo- rian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  2. [10]

    Barrier function-based safe reinforce- ment learning for emergency control of power systems

    Thanh Long Vu, Sayak Mukherjee, Renke Huang, and Qiuhua Huang. Barrier function-based safe reinforce- ment learning for emergency control of power systems. In2021 60th IEEE Conference on Decision and Control (CDC), pages 3652–3657. IEEE, 2021

    2021

  3. [11]

    Hyperparameters in reinforcement learning and how to tune them

    Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. InInternational Conference on Machine Learning, pages 9104–9149. PMLR, 2023

    2023

  4. [12]

    Safe policies for reinforce- ment learning via primal-dual methods.IEEE Transac- tions on Automatic Control, 68(3):1321–1336, 2022

    Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Cha- mon, and Alejandro Ribeiro. Safe policies for reinforce- ment learning via primal-dual methods.IEEE Transac- tions on Automatic Control, 68(3):1321–1336, 2022

    2022

  5. [13]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. InInterna- tional Conference on Machine Learning, pages 22–31. PMLR, 2017

    2017

  6. [14]

    Trust region-based safe distributional reinforcement learning for multiple constraints.Advances in Neural Information Processing Systems, 36, 2024

    Dohyeong Kim, Kyungjae Lee, and Songhwai Oh. Trust region-based safe distributional reinforcement learning for multiple constraints.Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    Ipo: Interior- point policy optimization under constraints

    Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior- point policy optimization under constraints. InProceed- ings of the AAAI conference on Artificial Intelligence, volume 34, pages 4940–4947, 2020

    2020

  8. [16]

    Crpo: A new approach for safe reinforcement learning with convergence guarantee

    Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. InInternational Conference on Machine Learning, pages 11480–11491. PMLR, 2021

    2021

  9. [17]

    Risk-averse model uncertainty for distributionally robust safe rein- forcement learning.Advances in Neural Information Processing Systems, 36, 2024

    James Queeney and Mouhacine Benosman. Risk-averse model uncertainty for distributionally robust safe rein- forcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    Constraints as rewards: Reinforcement learning for robots without reward functions.arXiv preprint arXiv:2501.04228, 2025

    Yu Ishihara, Noriaki Takasugi, Kotaro Kawakami, Masaya Kinoshita, and Kazumi Aoyama. Constraints as rewards: Reinforcement learning for robots without reward functions.arXiv preprint arXiv:2501.04228, 2025

    2025 arXiv

  11. [19]

    Safe and balanced: A framework for constrained multi-objective reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Alois Knoll, and Ming Jin. Safe and balanced: A framework for constrained multi-objective reinforcement learning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  12. [20]

    Safe distributed learning-enhanced predictive con- trol for multiple quadrupedal robots.arXiv preprint arXiv:2503.05836, 2025

    Weishu Zhan, Zheng Liang, Hongyu Song, and Wei Pan. Safe distributed learning-enhanced predictive con- trol for multiple quadrupedal robots.arXiv preprint arXiv:2503.05836, 2025

    2025 arXiv

  13. [21]

    Arthicha Srisuchinnawong and Poramate Manoonpong. An interpretable neural control network with adaptable online learning for sample efficient robot locomotion learning.IEEE Transactions on Neural Networks and Learning Systems, pages 1–13, 2025. doi: 10.1109/ TNNLS.2025.3552793

    2025

  14. [22]

    Hybrid reward architecture for reinforcement learning.Advances in Neural Information Processing Systems, 30, 2017

    Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Ro- main Laroche, Tavian Barnes, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning.Advances in Neural Information Processing Systems, 30, 2017

    2017

  15. [23]

    The general problem of the stability of motion.International Journal of Control, 55(3):531–534, 1992

    Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion.International Journal of Control, 55(3):531–534, 1992

    1992

  16. [24]

    Adaptive auxiliary task weighting for reinforce- ment learning.Advances in Neural Information Process- ing Systems, 32, 2019

    Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforce- ment learning.Advances in Neural Information Process- ing Systems, 32, 2019

    2019

  17. [25]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  18. [26]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. InConference on Robot Learning, pages 91–100. PMLR, 2022

    2022

  19. [27]

    Growable and interpretable neural control with online continual learning for autonomous lifelong locomotion learning machines.The International Journal of Robotics Research, 2025

    Arthicha Srisuchinnawong and Poramate Manoonpong. Growable and interpretable neural control with online continual learning for autonomous lifelong locomotion learning machines.The International Journal of Robotics Research, 2025. APPENDIX A. Comparison Table Table S1 presents ...

    2025

  20. [28]

    Consider the Lyapunov candidate function (R 1t), which measures the system’s deviation from equilibrium: V(R 1t) = 0.5R2 1t

    Stability Analysis:To present the stability of ROGER, this work adopts a Lyapunov-based approach [23]. Consider the Lyapunov candidate function (R 1t), which measures the system’s deviation from equilibrium: V(R 1t) = 0.5R2 1t. Given thatR 1t is a squared term,V(R 1t)is positi...

  21. [29]

    Learning Analysis:This study also proves that, while being stable (i.e., involving gentle system and learning dy- namics), adaptive adjustment of reward-weighting gains, as implemented in ROGER, ensures an increase in the primary rewardR 0t. Specifically, the cumulative change...

  22. [30]

    However, as shown in Figure S2, the tuning process is relatively complex

    Fixed-Gain Penalty:The fixed-gain penalty is a tradi- tional technique still employed in most works. However, as shown in Figure S2, the tuning process is relatively complex. On the one hand, low penalty weighting gains produce higher rewards, but the robot often falls during ...

  23. [31]

    Control Barrier Function (CBF):The CBF technique is also employed. However, as shown in Figure S3, although the CBF receives higher rewards than the fixed-gain penalty in general, most conditions still experience constraint violations during learning, while the tuning process ...

  24. [32]

    Primal-Dual Optimization (PDO):Figure S4 shows that PDO can prevent the robot from falling during training in some cases; however, it fails to meet the 10 ◦ constraint during learning in all cases and still requires careful hyperparameter tuning. Fig. S4: (left) Final primary ...

  25. [33]

    Online Learning with Auxiliary Loss (OL-AUX):Fig- ure S5 shows that, although OL-AUX can prevent the robot from falling during training in some cases, it fails to meet the 10◦ constraint and still requires careful hyperparameter tuning. Fig. S5: (left) Final primary reward ter...

  26. [34]

    Constraint-Rectified Parameter Optimization (CRPO): Figure S6 shows that, although CRPO can prevent the robot from falling during training in this case, it still requires careful selection of hyperparameter values to meet the constraints. Fig. S6: (left) Final primary reward t...

  27. [35]

    The figure shows that ROGER is less sensitive to the hyperparameter choices than other state-of-the-art methods since none of the tested cases cause the robot to fall

    Reward-Oriented Gains Via Embodied Regulation (ROGER):Given that the key hyperparameter of ROGER is the constraint threshold (τ i), Figure S7 presents the results after varyingτ 1 (roll constraint threshold) andτ 2 (pitch constraint threshold) between 5 ◦ and 17 ◦ (i.e., 0.1, ...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.