{"id":"ae5deb29-8481-4757-a55a-f5120fbda5b6","arxiv_id":"2505.07516","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"The authors' AR-EAPO controller achieves high simulated scores on swing-up tasks for acrobot and pendubot under increased disturbances by widening initial-state variance and shortening effective horizon during training.","lead":"This competition report tunes the authors' prior AR-EAPO reinforcement learning algorithm for acrobot and pendubot swing-up tasks in the 3rd AI Olympics, reporting high simulated scores under added external disturbances. It shows that adjusting the initial state distribution and episode truncation can make a model-free policy robust across random starts, though comparison baselines were not updated to the new rules.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central robustness claim is unverified: the paper never shows its simulation enforces the official 3rd AI Olympics max-velocity limit and disturbance model, and it admits baselines were not adapted to those rules.","rationale":"The reader identified the same load-bearing assumption: the simulation environment must reproduce the official evaluation protocol, including the revised maximum-velocity limit and disturbance model. My stress-test concurs, and adds that the paper's own admission about unadapted baselines strengthens the concern: if the authors did not adapt baselines to the new rules, it is unclear whether their own controller was evaluated under those rules either. The central claim of robust performance under the revised competition conditions cannot be accepted without verifying the environment match. This is a resolvable empirical issue, not a proof of failure, so the reader's CONDITIONAL verdict remains appropriate. No change to the verdict is needed, but the condition should be made explicit: the authors should show that the reported scores come from the official evaluation protocol, with code or checkpoints provided for reproducibility. I do not see a more fundamental mathematical inconsistency in the paper; the algorithmic description is plausible and builds on prior work, but the empirical validation is the weak link.","tokens_in":4924,"tokens_out":5666,"duration_ms":60620,"concrete_test":"Obtain the official 3rd AI Olympics RealAI Gym evaluation script (from the linked competition website or organizers), instantiate the exact max-velocity limit and disturbance model, and score the authors' policy on the same five seeds as Tables II and III. If the reproduced scores match within seed noise and no max-velocity violations occur, the concern is resolved; if scores differ or violations occur, the central claim must be revised or withdrawn. A minimal additional check is to instrument the evaluation environment to log whether any state exceeds the velocity limit during the reported trajectory plots.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AR-EAPO 'robustly manages' the revised tasks rests entirely on the reported RealAI Scores in Tables II and III. The paper does not state or demonstrate that its training/evaluation environment implements the official 3rd AI Olympics protocol, especially the revised maximum-velocity limit and disturbance model; the only link is a footnote to the competition website. This is not a minor omission because the reward design actively omits a torque penalty and uses a large reset variance to encourage fast swing-ups, which could produce velocities that exceed a newly introduced limit. The sentence in Section IV that baselines were 'tested in their original form, without modifications to reflect updated competition guidelines such as the revised maximum velocity limit' is doubly damaging: it makes the comparative claim unfair, and it raises the question of whether the authors' own controller was even evaluated under that limit. Without an explicit statement or log evidence that the max-velocity limit is enforced in the reported episodes, the absolute scores could reflect a private environment rather than the competition environment, so the central claim is not yet supported. The paper also provides no code or checkpoints, so this cannot be checked from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an extension of the authors' AR-EAPO reinforcement-learning algorithm to the acrobot and pendubot swing-up and stabilisation tasks under the updated guidelines of the 3rd AI Olympics (ICRA 2025). The proposed adaptation changes the MDP design rather than the policy optimisation: it uses a broad Gaussian initial-state distribution (σ=6.0), a large random truncation probability (ptrunc=5e-3), a relaxed quadratic cost without torque penalty, and reset noise variance 4.0. The authors report RealAI Scores over five seeds for acrobot (average 0.667) and pendubot (average 0.752), claiming that these are superior and more stable than several baselines, including MCPILCO, TVLQR, ILQR, MPC, ILQR Riccati, EVOL, SAC, and HISTORY-SAC.","tokens_in":5106,"tokens_out":3040,"duration_ms":30464,"significance":"If the empirical claims hold, the paper would demonstrate that a single model-free RL policy can robustly perform swing-up stabilisation of underactuated double pendulums under the competition's increased disturbances, which is a practically useful result for the AI Olympics setting. The paper gives credit to its concrete hyperparameter table, direct comparison with multiple external baselines, and the authors' prior competition record. However, the significance is currently limited by the absence of code, checkpoints, or environment validation, and the lack of statistical details (e.g., error bars, number of training runs, selection protocol), which prevents independent confirmation of the reported scores.","major_comments":[{"comment":"The manuscript never states or demonstrates that its simulation environment enforces the official 3rd AI Olympics evaluation protocol, especially the revised maximum-velocity limit and the disturbance/reset model; the only reference is a footnote to the competition website. This is load-bearing because the reward design explicitly omits a torque penalty and uses a large reset variance to encourage fast swing-ups, which could produce angular velocities exceeding the newly introduced limit. Without an explicit statement, log evidence, or code that the reported episodes were subject to the official limit, the absolute scores in Tables II and III cannot be attributed to the competition environment.","section":"Section IV, first paragraph and footnote 1"},{"comment":"The sentence 'Those controllers were tested in their original form, without modifications to reflect updated competition guidelines such as the revised maximum velocity limit' is doubly damaging: it makes the comparison to baselines unfair, because the baselines are handicapped by rules they were not adapted to, and it raises the question of whether the authors' own AR-EAPO controller was actually evaluated under that limit either. Therefore the central comparative claim, 'AR-EAPO consistently achieves superior and stable performance,' is not yet established under identical evaluation conditions.","section":"Section IV, paragraph on baseline testing"},{"comment":"The text says 'The best-performing policy from these runs was selected for reporting' after multiple training runs, but it provides no number of runs, no distribution of scores, and no correction for this selection. This makes the reported averages optimistically biased, and it contradicts the later claim that the results demonstrate 'stable' performance; at minimum, the manuscript must report the results of all runs or justify the selection rule and show that the reported seeds are representative.","section":"Section IV, policy selection paragraph"},{"comment":"The average-reward MaxEnt objective and the AR-EAPO advantage decomposition are described only sketchily, and the reader is referred to prior work for the algorithm's details. Since this report's contribution is presented as an adaptation of the MDP design rather than the learning algorithm, a self-contained description of how the estimated advantage, the gain update, and the PPO objective are computed is necessary for reproducibility; otherwise the empirical results cannot be reproduced from the manuscript alone.","section":"Section II, Eq. (1) and Section III"}],"minor_comments":[{"comment":"There are typos: 'specifcially' in the abstract and 'Opimization' in the conclusion; these should be corrected.","section":"Abstract and Conclusion"},{"comment":"Only the average over the five seeds is reported, without standard deviations or per-seed confidence intervals; adding these would make the 'stable' claim more credible.","section":"Tables II and III"},{"comment":"The captions state that these are the best-performing seeds; the text should explain why a representative seed was not used, or show all seeds, since the figures are used to demonstrate recovery from disturbances.","section":"Figures 1 and 2"},{"comment":"The definition of RealAI Score as 'total time the system remains within the designated goal region during each 60-second trial' should clarify the normalization that yields scores between 0 and 1, as reported in the tables.","section":"Section IV, first paragraph"},{"comment":"The initial state distribution is described as a Gaussian with σ=6.0, but since the angles are wrapped and the velocities are normalized, the meaning of this variance for each state coordinate should be specified more precisely.","section":"Section III"}],"recommendation":"major_revision","confidential_remarks":"This is a competition report that would benefit from an explicit statement of the official evaluation protocol and a clear commitment to sharing code/checkpoints. If the journal considers such empirical competition reports within scope, the paper may become acceptable after the environmental validation and fair comparison issues are addressed; if the scope prefers archival algorithmic contributions, the novelty may be limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: this is a competition report applying the authors' own AR-EAPO algorithm to the updated 3rd AI Olympics acrobot/pendubot tasks. What is new is not the algorithm (it comes from their prior papers) but a set of MDP design choices: Gaussian initial-state variance sigma=6, truncation probability 5e-3, no torque penalty, a relaxed quadratic cost, and reset noise variance 4.0. The absolute scores are plausible (acrobot ~0.67, pendubot ~0.75 across five seeds) and the hyperparameters are concrete.\n\nWhat the paper does well: it is transparent about the method's lineage, reports five-seed results rather than a single run, and explicitly admits the baselines were 'tested in their original form' without adapting to the new rules. That admission is honest, but it undercuts the comparative claim made in the conclusion ('consistently achieves superior and stable performance'). The baselines are not meaningful under the updated competition if they ignore the new max-velocity limit and disturbance model, and the paper never shows that its own policy was evaluated under that limit. The only link to the official protocol is a footnote. The stress-test note is right to flag this. It is not a fatal flaw, since the official environment is public and the authors likely ran in it, but a one-sentence statement confirming the official evaluation environment would close the gap.\n\nThere are also standard weaknesses: no code or data, no error bars or statistical tests, and the best policy was selected from multiple training runs. Those are minor for a competition report, but they matter when the claim is comparative.\n\nMy take: this is useful for the AI Olympics community as a record of what worked. The absolute results are probably usable, but the comparison is overstated. It deserves a serious referee if the authors clarify the evaluation environment and ideally release code or checkpoints. Without that, it is a borderline technical report.","headline":"A plausible competition report with an admitted-but-unaddressed unfair baseline comparison, and a missing statement that its own evaluation enforces the revised max-velocity limit.","tokens_in":5694,"tokens_out":2701,"would_cite":false,"duration_ms":27189,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using an average-reward maximum-entropy policy optimization algorithm called AR-EAPO, this paper claims to learn a single global controller that reliably swings up and stabilizes the acrobot and the pendubot under the 3rd AI Olympics'…","keywords":["average-reward reinforcement learning","maximum entropy","acrobot","pendubot","swing-up control","global policy","AR-EAPO","underactuated robotics"],"falsifier":"Run the trained AR-EAPO policy inside the official competition evaluation harness (or a faithful reimplementation) over more than five random seeds; if the average scores fall substantially below the reported 0.667 (acrobot) and 0.752 (pendubot), or if the controller fails to recover from disturbances specified by the official protocol, the claimed robustness is refuted. A more direct check is to inspect the simulation code to confirm that the implemented noise and reset process match the competition rules.","tokens_in":4662,"feed_emoji":"🤖","tokens_out":9283,"duration_ms":67017,"temperature":0.7,"pith_summary":"This paper claims that an average-reward maximum-entropy reinforcement learning algorithm, AR-EAPO, learns a single global controller for the acrobot and pendubot swing-up and stabilization tasks under the 3rd AI Olympics evaluation, which adds larger disturbances and a maximum-velocity limit. The authors argue that the key is to modify the MDP design rather than the learning algorithm: spread the initial state distribution widely, add a random truncation probability, and use a relaxed quadratic cost without a torque penalty. In simulations across five random seeds, the controller attains average scores of 0.667 on the acrobot and 0.752 on the pendubot, both above every tested baseline. A sympathetic reader would take the paper to be establishing that average-reward MaxEnt RL, with these environment-side adjustments, is a practical recipe for robust underactuated control.","feed_headline":"One policy swings up both acrobot and pendubot under noise","feed_subtitle":"Average-reward RL scores 0.667 and 0.752, beating earlier competition controllers.","key_machinery":"The central object is the AR-EAPO algorithm, which optimizes the average-reward maximum-entropy objective $\\rho^{\\pi}_{\\tau} = \\lim_{T\\to\\infty} \\frac{1}{T}\\mathbb{E}[\\sum_{t=0}^{T-1}(R_t - \\tau\\log\\pi(A_t|S_t))]$ for a unichain MDP. It defines a bias value function $v^{\\pi}_{\\tau}(s)$ satisfying a Bellman equation with the gain $\\rho^{\\pi}_{\\tau}$, then estimates an advantage function $a^{\\pi}(s,a) = r_{\\tau}(s,a) - \\rho^{\\pi}_{\\tau} + \\mathbb{E}_{s'}[v^{\\pi}_{\\tau}(s')] - v^{\\pi}_{\\tau}(s)$ by decomposing the bias into reward and entropy parts and applying generalized advantage estimation with different $\\lambda$ values to each. The policy network outputs a squashed Gaussian over torque, updated with the PPO objective. The paper's additional machinery is the MDP redesign: an initial-state Gaussian $\\mathcal{N}(s_0, \\sigma)$ with $\\sigma=6.0$, a random truncation probability $p_{\\text{trunc}}=0.005$, and the relaxed quadratic cost with scaling $\\alpha=0.001$ and $Q=\\mathrm{diag}([100,100,4,2])$.","core_discovery":"The central claim is that AR-EAPO, a model-free actor-critic method maximizing the long-run average reward with entropy regularization, transfers successfully to the updated competition setting without algorithmic changes, provided the training MDP is redesigned to mimic global conditions. The paper reports that this controller 'consistently achieves superior and stable performance' on both acrobot and pendubot, with score tables showing it beating prior competition entries under the same five seeds. The authors trace the success to three design choices: an initial-state Gaussian with standard deviation 6.0 centered at the downward position, a random truncation probability of 0.005 that shortens the effective horizon and favors faster swing-ups, and a cost function $r(s,a) = -\\alpha[(s-g)^{\\top}Q(s-g)]$ with no torque penalty. Because the average-reward objective has no discount factor, the policy optimizes the steady-state rate of being in the goal region rather than a discounted sum of rewards.","pith_inferences":["The comparison to baselines is not fully apples-to-apples: the baseline controllers were tested in their original form without adjusting for the new maximum-velocity limit, so part of the margin may reflect rule changes rather than algorithmic superiority.","The random truncation probability is effectively a horizon control knob; the same trick could be applied to other average-reward or continuing-task RL settings to bias policies toward short-horizon objectives.","Because the paper reports only five seeds and picks the best policy from multiple training runs, the stability claim would be strengthened by reporting variance across more seeds and across training runs.","The relaxed reward (no torque penalty) is feasible because the evaluation rewards time-in-goal, not energy efficiency; for real robots, an energy penalty would likely still be needed."],"forward_implications":["If the paper is correct, the trained AR-EAPO controllers can serve as drop-in global swing-up policies for acrobot and pendubot under the updated competition evaluation, holding the upright position for most of a 60-second trial despite periodic disturbances.","The reported performance implies that the average-reward MaxEnt objective, without discounting, is a viable alternative for continuing control tasks where episode boundaries are artificial.","The MDP design choices (wide initial-state variance, random truncation probability, relaxed cost) appear sufficient to induce global behavior without reward shaping or explicit system modeling.","The superiority over prior competition controllers suggests that adapting the training environment to the evaluation protocol matters more than changing the underlying RL algorithm."],"supporting_citations":[{"why":"Defines the AR-EAPO algorithm whose average-reward MaxEnt objective drives the controller.","marker":"[6]"},{"why":"Earlier AR-EAPO version that the authors refine; supplies the random truncation mechanism.","marker":"[5]"},{"why":"Generalized advantage estimation, used for separate variance reduction of reward and entropy parts.","marker":"[7]"},{"why":"PPO objective used for policy updates.","marker":"[8]"},{"why":"Defines the acrobot/pendubot platform and evaluation metric.","marker":"[2]"},{"why":"Reports the 2nd AI Olympics results, furnishing the baseline controllers compared here.","marker":"[4]"},{"why":"MCPILCO baseline controller tested for comparison.","marker":"[9]"},{"why":"Evolutionary SAC baseline controller.","marker":"[10]"},{"why":"Velocity-history SAC baseline controller.","marker":"[11]"}],"fun_headline_variants":["One policy swings up both acrobot and pendubot","Average-reward RL tops prior double-pendulum controllers","RL without discount stabilizes acrobot and pendubot","AR-EAPO: single policy for acrobot and pendubot","Global RL policy outperforms earlier pendulum controllers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusions stand on the assumption that its simulation environment reproduces the official competition evaluation, including the revised maximum-velocity limit and the disturbance model, a match that is asserted only by pointing to the competition website and never validated in the text.","fun_headline_variants_meta":{"raw":{"variants":["One policy swings up both acrobot and pendubot","Average-reward RL tops prior double-pendulum controllers","RL without discount stabilizes acrobot and pendubot","AR-EAPO: single policy for acrobot and pendubot","Global RL policy outperforms earlier pendulum controllers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001229,"raw_usage":{"total_tokens":4988,"prompt_tokens":822,"completion_tokens":4166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":4085}},"tokens_in":438,"tokens_out":4166,"duration_ms":31046,"temperature":1.0,"reasoning_tokens":4085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:14:28.560776+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained AR-EAPO policy inside the official competition evaluation harness (or a faithful reimplementation) over more than five random seeds; if the average scores fall substantially below the reported 0.667 (acrobot) and 0.752 (pendubot), or if the controller fails to recover from disturbances specified by the official protocol, the claimed robustness is refuted. A more direct check is to inspect the simulation code to confirm that the implemented noise and reset process match the competition rules.","supporting_citations":[{"cited_title":"Wiebe, S","cited_arxiv_id":null,"evidence_quote":"Defines the acrobot/pendubot platform and evaluation metric."},{"cited_title":"AI Olympics challenge with Evolutionary Soft Actor Critic","cited_arxiv_id":"2409.01104","evidence_quote":"Velocity-history SAC baseline controller."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the AR-EAPO algorithm whose average-reward MaxEnt objective drives the controller."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Generalized advantage estimation, used for separate variance reduction of reward and entropy parts."}],"review_version":1}