REVIEW 4 major objections 5 minor 11 references
Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Using an average-reward maximum-entropy policy optimization algorithm called AR-EAPO, this paper claims to learn a single global controller that reliably swings up and stabilizes the acrobot and the pendubot under the 3rd AI Olympics'…
desk verdict A plausible competition report with an admitted-but-unaddressed unfair baseline comparison, and a missing statement that its own evaluation enforces the revised max-velocity limit. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the AR-EAPO algorithm, which optimizes the average-reward maximum-entropy objective $\rho^{\pi}_{\tau} = \lim_{T\to\infty} \frac{1}{T}\mathbb{E}[\sum_{t=0}^{T-1}(R_t - \tau\log\pi(A_t|S_t))]$ for a unichain MDP. It defines a bias value function $v^{\pi}_{\tau}(s)$ satisfying a Bellman equation with the gain $\rho^{\pi}_{\tau}$, then estimates an advantage function $a^{\pi}(s,a) = r_{\tau}(s,a) - \rho^{\pi}_{\tau} + \mathbb{E}_{s'}[v^{\pi}_{\tau}(s')] - v^{\pi}_{\tau}(s)$ by decomposing the bias into reward and entropy parts and applying generalized advantage estimation with different $\lambda$ values to each. The policy network outputs a squashed Gaussian over torque, updated with the PPO objective. The paper's additional machinery is the MDP redesign: an initial-state Gaussian $\mathcal{N}(s_0, \sigma)$ with $\sigma=6.0$, a random truncation probability $p_{\text{trunc}}=0.005$, and the relaxed quadratic cost with scaling $\alpha=0.001$ and $Q=\mathrm{diag}([100,100,4,2])$.
What would settle it
Run the trained AR-EAPO policy inside the official competition evaluation harness (or a faithful reimplementation) over more than five random seeds; if the average scores fall substantially below the reported 0.667 (acrobot) and 0.752 (pendubot), or if the controller fails to recover from disturbances specified by the official protocol, the claimed robustness is refuted. A more direct check is to inspect the simulation code to confirm that the implemented noise and reset process match the competition rules.
Extended reading notes
Core claim
The central claim is that AR-EAPO, a model-free actor-critic method maximizing the long-run average reward with entropy regularization, transfers successfully to the updated competition setting without algorithmic changes, provided the training MDP is redesigned to mimic global conditions. The paper reports that this controller 'consistently achieves superior and stable performance' on both acrobot and pendubot, with score tables showing it beating prior competition entries under the same five seeds. The authors trace the success to three design choices: an initial-state Gaussian with standard deviation 6.0 centered at the downward position, a random truncation probability of 0.005 that shortens the effective horizon and favors faster swing-ups, and a cost function $r(s,a) = -\alpha[(s-g)^{\top}Q(s-g)]$ with no torque penalty. Because the average-reward objective has no discount factor, the policy optimizes the steady-state rate of being in the goal region rather than a discounted sum of rewards.
Load-bearing premise
The paper's conclusions stand on the assumption that its simulation environment reproduces the official competition evaluation, including the revised maximum-velocity limit and the disturbance model, a match that is asserted only by pointing to the competition website and never validated in the text.
Editorial extensions
If this is right
- If the paper is correct, the trained AR-EAPO controllers can serve as drop-in global swing-up policies for acrobot and pendubot under the updated competition evaluation, holding the upright position for most of a 60-second trial despite periodic disturbances.
- The reported performance implies that the average-reward MaxEnt objective, without discounting, is a viable alternative for continuing control tasks where episode boundaries are artificial.
- The MDP design choices (wide initial-state variance, random truncation probability, relaxed cost) appear sufficient to induce global behavior without reward shaping or explicit system modeling.
- The superiority over prior competition controllers suggests that adapting the training environment to the evaluation protocol matters more than changing the underlying RL algorithm.
Reading between the lines
- The comparison to baselines is not fully apples-to-apples: the baseline controllers were tested in their original form without adjusting for the new maximum-velocity limit, so part of the margin may reflect rule changes rather than algorithmic superiority.
- The random truncation probability is effectively a horizon control knob; the same trick could be applied to other average-reward or continuing-task RL settings to bias policies toward short-horizon objectives.
- Because the paper reports only five seeds and picks the best policy from multiple training runs, the stability claim would be strengthened by reporting variance across more seeds and across training runs.
- The relaxed reward (no torque penalty) is feasible because the evaluation rewards time-in-goal, not energy efficiency; for real robots, an energy penalty would likely still be needed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an extension of the authors' AR-EAPO reinforcement-learning algorithm to the acrobot and pendubot swing-up and stabilisation tasks under the updated guidelines of the 3rd AI Olympics (ICRA 2025). The proposed adaptation changes the MDP design rather than the policy optimisation: it uses a broad Gaussian initial-state distribution (σ=6.0), a large random truncation probability (ptrunc=5e-3), a relaxed quadratic cost without torque penalty, and reset noise variance 4.0. The authors report RealAI Scores over five seeds for acrobot (average 0.667) and pendubot (average 0.752), claiming that these are superior and more stable than several baselines, including MCPILCO, TVLQR, ILQR, MPC, ILQR Riccati, EVOL, SAC, and HISTORY-SAC.
Significance. If the empirical claims hold, the paper would demonstrate that a single model-free RL policy can robustly perform swing-up stabilisation of underactuated double pendulums under the competition's increased disturbances, which is a practically useful result for the AI Olympics setting. The paper gives credit to its concrete hyperparameter table, direct comparison with multiple external baselines, and the authors' prior competition record. However, the significance is currently limited by the absence of code, checkpoints, or environment validation, and the lack of statistical details (e.g., error bars, number of training runs, selection protocol), which prevents independent confirmation of the reported scores.
major comments (4)
- [Section IV, first paragraph and footnote 1] The manuscript never states or demonstrates that its simulation environment enforces the official 3rd AI Olympics evaluation protocol, especially the revised maximum-velocity limit and the disturbance/reset model; the only reference is a footnote to the competition website. This is load-bearing because the reward design explicitly omits a torque penalty and uses a large reset variance to encourage fast swing-ups, which could produce angular velocities exceeding the newly introduced limit. Without an explicit statement, log evidence, or code that the reported episodes were subject to the official limit, the absolute scores in Tables II and III cannot be attributed to the competition environment.
- [Section IV, paragraph on baseline testing] The sentence 'Those controllers were tested in their original form, without modifications to reflect updated competition guidelines such as the revised maximum velocity limit' is doubly damaging: it makes the comparison to baselines unfair, because the baselines are handicapped by rules they were not adapted to, and it raises the question of whether the authors' own AR-EAPO controller was actually evaluated under that limit either. Therefore the central comparative claim, 'AR-EAPO consistently achieves superior and stable performance,' is not yet established under identical evaluation conditions.
- [Section IV, policy selection paragraph] The text says 'The best-performing policy from these runs was selected for reporting' after multiple training runs, but it provides no number of runs, no distribution of scores, and no correction for this selection. This makes the reported averages optimistically biased, and it contradicts the later claim that the results demonstrate 'stable' performance; at minimum, the manuscript must report the results of all runs or justify the selection rule and show that the reported seeds are representative.
- [Section II, Eq. (1) and Section III] The average-reward MaxEnt objective and the AR-EAPO advantage decomposition are described only sketchily, and the reader is referred to prior work for the algorithm's details. Since this report's contribution is presented as an adaptation of the MDP design rather than the learning algorithm, a self-contained description of how the estimated advantage, the gain update, and the PPO objective are computed is necessary for reproducibility; otherwise the empirical results cannot be reproduced from the manuscript alone.
minor comments (5)
- [Abstract and Conclusion] There are typos: 'specifcially' in the abstract and 'Opimization' in the conclusion; these should be corrected.
- [Tables II and III] Only the average over the five seeds is reported, without standard deviations or per-seed confidence intervals; adding these would make the 'stable' claim more credible.
- [Figures 1 and 2] The captions state that these are the best-performing seeds; the text should explain why a representative seed was not used, or show all seeds, since the figures are used to demonstrate recovery from disturbances.
- [Section IV, first paragraph] The definition of RealAI Score as 'total time the system remains within the designated goal region during each 60-second trial' should clarify the normalization that yields scores between 0 and 1, as reported in the tables.
- [Section III] The initial state distribution is described as a Gaussian with σ=6.0, but since the angles are wrapped and the velocities are normalized, the meaning of this variance for each state coordinate should be specified more precisely.
Circularity Check
No significant circularity: the central claim is an empirical benchmark result against external baselines, not a derivation from the algorithm's own equations.
full rationale
The paper makes no first-principles derivation claim; it reports an empirical comparison of a previously published algorithm (AR-EAPO) adapted to a competition benchmark. The load-bearing evidence is the measured RealAI Score against external baselines (MCPILCO, TVLQR, ILQR, MPC, EVOL SAC, HISTORY-SAC), not an equation that reduces to a fitted parameter. Hyperparameters such as noise variance, ptrunc, and reward weights are training choices, but the reported scores are measured on the environment and are not algebraically implied by those choices. Self-citations [5] and [6] only identify the algorithm and prior competition results; the present performance claim stands or falls on external comparisons and simulation outcomes, so the self-citation is not load-bearing. The admitted fact that baseline controllers were not updated to the revised maximum-velocity limit (Section IV) weakens the comparative claim, but that is a fairness/validity limitation, not circularity. No quoted step exhibits Eq. X = Eq. Y by construction or a fitted value renamed as prediction; hence no significant circularity is present.
Assumptions & free parameters
free parameters (7)
- sigma=6.0 =
6.0
- ptrunc=5e-3 =
5e-3
- alpha=0.001 =
0.001
- Q=diag([100,100,4,2]) =
[100,100,4,2]
- reset noise variance=4.0 =
4.0
- temperature tau=1.5 =
1.5
- network sizes =
policy [256,256], value [512,512]
assumptions (4)
- domain assumption Unichain MDP assumption for the average-reward formulation
- standard math Average-reward Bellman equation and bias value function
- domain assumption The acrobot and pendubot dynamics follow the RealAI Gym simulator
- domain assumption Squashed Gaussian policy class is sufficiently expressive
Cite this review
Pith. "Pith review of Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks." pith.science (2026). https://pith.science/paper/OVHCJWN3
@misc{pith2026250507516,
author = {Pith},
title = {Pith review of: Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks},
year = {2026},
howpublished = {\url{https://pith.science/paper/OVHCJWN3}},
note = {Machine review of arXiv:2505.07516}
}
read the original abstract
This report presents our reinforcement learning-based approach for the swing-up and stabilisation tasks of the acrobot and pendubot, tailored specifcially to the updated guidelines of the 3rd AI Olympics at ICRA 2025. Building upon our previously developed Average-Reward Entropy Advantage Policy Optimization (AR-EAPO) algorithm, we refined our solution to effectively address the new competition scenarios and evaluation metrics. Extensive simulations validate that our controller robustly manages these revised tasks, demonstrating adaptability and effectiveness within the updated framework.
Figures
Reference graph
Works this paper leans on
-
[2]
F. Wiebe, S. Vyas, L. Maywald, S. Kumar, and F. Kirchner, ``Realaigym: Education and research platform for studying athletic intelligence,'' in Proceedings of Robotics Science and Systems Workshop Mind the Gap: Opportunities and Challenges in the Transition Between Research and Industry, New York, 2022
work page 2022
- [3]
-
[4]
ai olympics with realaigym
F. Wiebe, N. Turcato, A. Dalla Libera, C. Zhang, T. Vincent, S. Vyas, G. Giacomuzzo, R. Carli, D. Romeres, A. Sathuluri, M. Zimmermann, B. Belousov, J. Peters, F. Kirchner, and S. Kumar, ``Reinforcement learning for athletic intelligence: Lessons from the 1st “ai olympics with realaigym” competition,'' in Proceedings of the Thirty-Third International Join...
2024
-
[5]
F. Wiebe, N. Turcato, A. D. Libera, J. S. B. Choe, B. Choi, T. L. Faust, H. Maraqten, E. Aghadavoodi, M. Cali, A. Sinigaglia et al., ``Reinforcement learning for robust athletic intelligence: Lessons from the 2nd'ai olympics with realaigym'competition,'' arXiv preprint arXiv:2503.15290, 2025
arXiv 2025
-
[6]
J. S. Bjorn Choe, B. Choi, and J.-k. Kim, ``Average-reward maximum entropy reinforcement learning for underactuated double pendulum tasks,'' arXiv e-prints, pp. arXiv--2409, 2024
work page 2024
-
[7]
J. S. B. Choe, B. Choi, and J.-k. Kim, ``Simplifying reward design in complex robotics:\\ average-reward maximum entropy reinforcement learning,'' in 2025 IEEE International Conference on Robotics and Automation (ICRA). 1em plus 0.5em minus 0.4em IEEE, 2025
work page 2025
-
[8]
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, ``High-dimensional continuous control using generalized advantage estimation,'' arXiv preprint arXiv:1506.02438, 2015
arXiv 2015
-
[9]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, ``Proximal policy optimization algorithms,'' arXiv preprint arXiv:1707.06347, 2017
arXiv 2017
Show all 11 references
-
[10]
Turcato, A
N. Turcato, A. D. Libera, G. Giacomuzzo, R. Carli, and D. Romeres, ``Learning control of underactuated double pendulum with model-based reinforcement learning,'' arXiv preprint arXiv:2409.05811, 2024
2024 arXiv
-
[11]
Cal \` , A
M. Cal \` , A. Sinigaglia, N. Turcato, R. Carli, and G. A. Susto, ``Ai olympics challenge with evolutionary soft actor critic,'' arXiv preprint arXiv:2409.01104, 2024
2024 arXiv
-
[12]
ai olympics with realaigym
T. Lukas Faust, H. Maraqten, E. Aghadavoodi, B. Belousov, and J. Peters, ``Velocity-history-based soft actor-critic tackling iros'24 competition" ai olympics with realaigym",'' arXiv e-prints, pp. arXiv--2410, 2024
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.