Pith. sign in

REVIEW 4 major objections 5 minor 56 references

In a large-scale study of actor-critic components on a realistic water-treatment control task, the paper argues that common defaults—Gaussian policies with pathwise gradient estimators and fixed critic update ratios—are among the least reli

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:37 UTC pith:2HNFH5IF

load-bearing objection Useful large-scale component-level study, but the adaptive-update headline is a bandit result stretched to MDPs. the 4 major comments →

arxiv 2607.13274 v1 pith:2HNFH5IF submitted 2026-07-14 cs.LG cs.AI

Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

Haseeb Shah , Lingwei Zhu , Adam White , Martha White This is my paper
classification cs.LG cs.AI
keywords reinforcement learningactor-criticalgorithm reliabilitypolicy parameterizationbeta policygradient estimatorsupdate-to-data ratioPID control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the reliability of actor-critic reinforcement learning—measured as run-to-run variability and sensitivity to hyperparameters—is governed more by a handful of component choices than by which named algorithm you select. Using a non-contextual bandit task derived from a real drinking-water backwash pump, it runs over 33,000 experiments and finds that the popular defaults are often the worst: Gaussian action distributions with pathwise gradient estimators are among the least reliable configurations, whereas bounded distributions (beta policies) with adaptive critic update schedules remain robust. The paper traces most of the instability to critic inaccuracy: when the learned value estimate is wrong, the actor inherits that error at every step. It therefore argues for mixing observed rewards into the policy gradient, using larger critics, and replacing fixed critic update-to-data ratios with an adaptive error-threshold scheme. The paper also flags that several of these conclusions rest on per-step mechanisms that "should transfer" to full sequential MDP settings, but that transfer is not itself demonstrated.

Core claim

On the paper's own terms, the central discovery is a ranking of design choices by reliability rather than best-case reward. Beta parameterizations occupy the best region of the performance-stability frontier across five algorithms; the squashed Gaussian policy that is the standard default in continuous-control SAC sits in a strictly dominated region, and Gaussian and Student's-t policies with pathwise gradients degrade badly at higher entropies, especially when clipping or squashing distorts the sampled action path. Higher actor update-to-data ratios help beta policies (higher mean reward, lower variability) while destabilizing Gaussian policies, whose variability can quadruple. An adaptive

What carries the argument

The central objects are the policy parameterization and the path through which gradient information reaches the actor. A beta policy encodes the action-space bounds in its density (so there is no clipped or squashed path for the pathwise gradient to flow through), whereas Gaussian and Student's-t policies are typically clipped or squashed, creating regions where pathwise gradients vanish or distort. A second mechanism is the adaptive critic update-to-data (UTD) ratio: instead of a fixed number of critic updates per environment step, Algorithm 1 updates the critic until its batch mean-squared error falls below a threshold ω, keeping the actor update coupled to a measured level of critic accur

Load-bearing premise

The practical advice collapses if the mechanisms observed in the stateless bandit PID setting—where the critic is fit by regression on immediate rewards—do not transfer to full MDPs with bootstrapped targets, state-dependent exploration, and replay distribution shift; the paper itself flags this transfer as an assumption ("should transfer") rather than a demonstrated result.

What would settle it

A focused MDP experiment: hold the algorithm fixed (e.g., SAC) on a continuous-control benchmark and compare the paper's recommended configuration (beta policy, likelihood-ratio or matched gradient, adaptive critic UTD) against the default (squashed Gaussian, pathwise, fixed critic UTD). If the recommended configuration fails to achieve lower run-to-run coefficient of variation than the default across seeds and learning rates, the headline claim is contradicted. Equivalently, a single result showing that raising actor UTD improves a Gaussian policy (rather than destabilizing it) would falsify

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners should treat Gaussian policies with pathwise gradients as a choice to verify, not a safe default, especially when the policy is clipped or squashed.
  • Beta policies with elevated actor UTD ratios are a robust configuration across algorithms, improving mean reward and lowering variability.
  • Adaptive critic updating can replace the sensitive fixed critic UTD hyperparameter with an interpretable error threshold.
  • Switching the policy parameterization within an algorithm can change outcomes more than switching between algorithms.
  • The bandit-derived mechanisms (gradient distortion, adaptive critic benefit) are predicted to carry over to MDP settings, and are falsifiable there.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the bandit findings transfer, existing continuous-control codebases that default to squashed-Gaussian policies with pathwise gradients and fixed critic UTD ratios would be expected to show immediate reliability gains from three local changes: a beta policy, a likelihood-ratio gradient option, and an adaptive critic update.
  • Editorial inference: the paper's mechanism suggests that in MDPs the interaction between high actor UTD and critic tracking of a fast-changing actor could be even more delicate, because the actor changes the critic's bootstrapping target; a useful next experiment is probing actor UTD x critic UTD jointly on a canonical continuous-control benchmark.
  • Editorial inference: the reward-mixing result hints at a simple robustness retrofit for off-policy algorithms as well—using λ-return or a small mixing of observed returns—though the paper notes this raises variance with off-policy data.
  • Editorial inference: a testable extension is to run the same component sweep on a small set of stationary and non-stationary MDP control tasks; the paper predicts beta+adaptive will keep its low coefficient of variation, and identifying which components dominate under non-stationarity would be practically valuable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a large empirical study (33,000+ runs) of actor-critic design components in a non-contextual bandit setting built from a real water-treatment PID control task. The authors systematically vary policy parameterization (Gaussian, beta, Student's t, squashed/clipped variants), gradient estimator (likelihood-ratio vs pathwise), actor update objective (AC(λ), SAC, PPO, MPO, GreedyAC, DDPG, mirror-descent SAC), actor and critic update-to-data ratios, and critic capacity. The main reported findings are: mixing observed rewards into the actor update improves stability; critic inaccuracy drives run-to-run variability; beta policies achieve the best performance-variability frontier; bounded policies converge slower; actor UTD is a safer tuning knob than actor learning rate; higher actor UTD helps beta but hurts Gaussian policies; pathwise gradients are unreliable with clipped/squashed policies; and an adaptive critic-updating rule (Algorithm 1) is more reliable than fixed critic UTD ratios. The paper is explicitly framed as guidance for practitioners choosing actor-critic components in real-world control, and it states that several conclusions should transfer to full MDP settings, while also acknowledging that the MDP stopping signal is only a surrogate for value error.

Significance. If the findings hold, the paper would provide a useful component-level decomposition of actor-critic methods and a concrete counterpoint to common defaults (e.g., squashed Gaussian policies with pathwise gradients). The strengths include a reproducible experimental protocol (released code and data), bootstrap CIs, a real-data-inspired environment, and a clear set of falsifiable conclusions with explicit limitations. The study is particularly valuable in drawing attention to run-to-run variability and hyperparameter sensitivity as first-class metrics, rather than mean performance alone. The main caveat is scope: all experiments are in a single bandit environment, and the adaptive-critic-UTD conclusion is tied to a stopping rule that directly measures reward regression error, which does not have the same meaning under TD learning in MDPs.

major comments (4)
  1. [Discussion, 'Implications beyond the bandit setting'; Materials and Methods, Algorithm 1] All headline conclusions are obtained in a non-contextual bandit, and the adaptive-critic-UTD result in particular rests on a stopping rule that measures the regression error between q_w(a) and the observed reward (Alg. 1, line 7). In an MDP the analogous stopping signal would be the temporal-difference error, which the paper itself states is 'a surrogate for the value error.' TD error can be small when the value estimate is systematically wrong (e.g., under bootstrapping, target lag, or distribution shift), so a threshold on TD error does not guarantee the critic is accurate before each actor update. Since Alg. 1 is the default critic update for every main sweep ('Unless otherwise stated...'), the abstract's claim that 'bounded distributions with adaptive update schedules remain robust' is currently established only for the stateless case. The paper's 'Future directions' explicitly says
  2. [Fig. 4; Materials and Methods, Evaluation Criteria] The performance-stability frontier (Fig. 4) is constructed by first selecting the top-10 hyperparameter configurations by mean reward on the same 10-seed runs and then computing aggregate mean and mean CV on those runs. This is selection on the dependent variable: the reported means are maxima over the grid and are biased upward, and the CVs of selected configurations are not representative of the full configuration distribution. Bootstrapping the selected runs does not correct this. Fig. 4 is the main evidence for Conclusion 3 and for the abstract's statement about bounded distributions being robust; without a held-out evaluation of the selected configurations (or a full-sweep frontier), the claim is not supported at the stated strength.
  3. [Experimental Setup; Backwashing-PID Environment] All experiments use a single fitted environment: the reward is computed from one second-order polynomial (Eq. 2) with fixed noise levels, action bounds, and setpoint (Table S1). The only environmental variation is reward noise in S8 and critic capacity; the plant dynamics and reward surface are never changed. Therefore the 'wide range of settings' in the abstract refers to hyperparameters and algorithms within one bandit task, not to a range of environments. Since the paper's stated goal includes guiding deployment to 'new real-world control settings,' at least one independent bandit task with a different reward surface would materially strengthen the generality claim; otherwise the language should be narrowed.
  4. [Fig. 9; Experimental Setup] Several quantitative claims about variability rest on 10 seeds per configuration. For example, Conclusion 8 states that UTD_critic=1 produces '11x higher run-to-run variability' for Gaussian policies under UTD_actor=10. With N=10, the coefficient of variation itself has a large sampling error, and a ratio of two CV estimates is highly unstable. The bootstrap CIs in Fig. 9 are on the mean reward gap, not on the CV ratio. Please report CIs for the CV ratios or aggregate over more seeds for the specific configurations used in the headline comparisons.
minor comments (5)
  1. [Abstract] The phrase 'Gaussian action distributions with pathwise gradient estimators' conflates clipped Gaussian and squashed Gaussian policies. The plot-level evidence shows different behavior for these variants; please specify which variant is meant.
  2. [Materials and Methods] The text says rewards are reported as -log10(|Reward|) while figure axes label 'log10(Reward)'. State the transformation explicitly in each figure caption to avoid confusion.
  3. [Throughout] The abbreviation 'UTD' is defined as 'update-to-data' in one place and 'update-to-date' in another. Please use one consistent definition.
  4. [Algorithm 1] The symbol B is used both for the replay buffer and for the mini-batch in lines 5-7. Use a different symbol for the mini-batch to improve readability.
  5. [Algorithm section, PPO] It may be worth reminding readers that in the bandit setting PPO uses no learned critic, whereas in MDPs PPO typically learns a state-value critic. The Discussion acknowledges this, but an explicit note in the algorithm section would help.

Circularity Check

0 steps flagged

No significant circularity: the study's empirical conclusions are self-contained comparisons, not fitted parameters relabeled as predictions.

full rationale

The paper is a large-scale empirical study, not a derivation. Its conclusions (e.g., beta policies more robust, adaptive critic UTD more reliable, pathwise estimators problematic with clipped/squashed policies) are direct observations from sweeps over component choices with fixed seeds and hyperparameters. Algorithm 1's adaptive critic update is an evaluated method, not a fitted parameter disguised as a prediction; the comparison against fixed UTD ratios in Fig 9 is an empirical head-to-head. Self-citations (GreedyAC [17], mirror-descent SAC [26], KL greedification unification [24]) provide background algorithms and framing, but none is load-bearing for the central empirical claims, and no cited result is invoked to forbid alternatives. The bandit-to-MDP transfer is explicitly conceded as untested ('The most direct next step is to test which of our conclusions transfer to the standard MDP setting'), so it is an external-validity limitation, not a circular step. No equation in the paper reduces to its own inputs.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The central claims rest on a simulated environment whose polynomial model, noise levels, reward normalization, and action bounds are chosen or fitted by the authors, plus the untested assumption that bandit results transfer to MDPs. No new physical entities are introduced.

free parameters (9)
  • Flow model polynomial coefficients (Eq. 2) = a=6.0553e-4, b=-2.530755e-2, c=0.26727398
    Fitted to empirical pump speed/flow data from the Drayton Valley plant; defines the reward surface.
  • Flow setpoint f_ideal = 0.6309013
    Chosen from plant data; the target the controller must track.
  • Measurement noise std σ_f = 0.003
    Fixed by hand in all experiments; affects stochasticity of the environment.
  • Reward noise std σ_r = 0.01
    Fixed by hand; also swept in one experiment (0, 0.01, 0.1, 0.3).
  • Reward deviation divisor = 110
    Chosen to normalize rewards to roughly [-1, 0].
  • Action range [p,i,d] = [0,20]
    Chosen action bounds; central to the bounded vs clipped/squashed comparison.
  • Adaptive critic error tolerance ω = 0.001
    Chosen threshold in Algorithm 1; central to Conclusion 8.
  • Adaptive critic max updates n_steps = 100
    Ceiling on critic updates per step in Algorithm 1.
  • Policy initializations = Gaussian μ=0 σ=1; Beta α=5 β=5; Student-t ν=3
    Chosen by hand; initialization can influence convergence and variability.
axioms (5)
  • domain assumption The second-order polynomial in Eq. 2 adequately represents the real water-treatment pump dynamics.
    Invoked in Materials and Methods; all conclusions depend on the simulator.
  • domain assumption A stateless bandit PID-tuning task preserves the component interactions that determine actor-critic reliability in full MDPs.
    The paper restricts to the bandit setting and argues transfer by mechanism in the Discussion.
  • ad hoc to paper The squared-error threshold ω on the critic batch loss is a good proxy for critic accuracy across environments.
    Algorithm 1 and Conclusion 8 rely on this interpretability claim.
  • domain assumption Trends identified on the top-10 hyperparameter configurations selected by mean reward generalize to the full hyperparameter space.
    Figure 4 and related aggregations use this selection; it may bias CV estimates.
  • standard math The policy-gradient estimators (likelihood-ratio and pathwise) are unbiased Monte Carlo estimates under standard smoothness conditions.
    Standard background invoked in the Gradient Estimators section.

pith-pipeline@v1.3.0-alltime-deepseek · 35271 in / 10408 out tokens · 101245 ms · 2026-08-02T05:37:55.732501+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.

Figures

Figures reproduced from arXiv: 2607.13274 by Adam White, Haseeb Shah, Lingwei Zhu, Martha White.

Figure 1
Figure 1. Figure 1: Schematic of a reinforcement learning system on a water treatment controller [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mixing immediate reward with value estimates improves stability and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance-stability frontier across algorithms and policy parame [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Bounded-support policies converge slower than unbounded-support across algorithms [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Actor update-to-data ratio is a safer tuning knob compared to the actor learning rate. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Higher actor update-to-data ratio improves beta policies but destabilizes Gaussian policies. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pathwise gradient estimators are often unreliable with clipped and squashed policy parameterizations. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Adaptive critic updating is more reliable than a fixed critic update-to-data ratio. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 1 canonical work pages

  1. [1]

    Mastering the game of Go without human knowledge.Nature, 550:354–359, 2017

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge.Nature, 550:354–359, 2017. 11 of 38

    2017

  2. [2]

    Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning.Nature, 610:47–53, 2022

    2022

  3. [3]

    Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Sub- ramanian, Thomas J

    Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Sub- ramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Ro...

    2022

  4. [4]

    Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602:414–419, 2022

    Jonas Degrave, Federico Felici, and et al. Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602:414–419, 2022

    2022

  5. [5]

    Dense reinforcement learning for safety validation of autonomous vehicles.Nature, pages 620–627, 2023

    Shuo Feng, Haowei Sun, and et al. Dense reinforcement learning for safety validation of autonomous vehicles.Nature, pages 620–627, 2023

    2023

  6. [6]

    A survey of reinforcement learning for software engineering, 2025

    Dong Wang, Hanmo Y ou, Lingwei Zhu, Kaiwei Lin, Zheng Chen, Chen Y ang, Junji Yu, Zan Wang, and Junjie Chen. A survey of reinforcement learning for software engineering, 2025

    2025

  7. [7]

    Rainbow: Combining im- provements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  8. [8]

    Noisy networks for exploration

    Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. InInternational Confer- ence on Learning Representations, 2018

    2018

  9. [9]

    A distributional perspective on rein- forcement learning

    Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on rein- forcement learning. InInternational conference on machine learning, pages 449–458. Pmlr, 2017

    2017

  10. [10]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    Pith/arXiv arXiv 2015

  11. [11]

    The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    Marc G Bellemare, Y avar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents.Journal of artificial intelligence research, 47:253–279, 2013

    2013

  12. [12]

    Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research

    Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. InInternational Conference on Machine Learning, pages 1373–1383. PMLR, 2021

    2021

  13. [13]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InProceedings of the 35th International Conference on Machine Learning, pages 1861–1870, 2018

    2018

  14. [14]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  15. [15]

    Deepmind control suite

    Yuval Tassa, Y otam Doron, Alistair Muldal, Tom Erez, Y azhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    Pith/arXiv arXiv 2018

  16. [16]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. InInternational Conference on Learning Representations, 2018

    2018

  17. [17]

    Greedy actor-critic: A new conditional cross-entropy method for policy im- provement

    Samuel Neumann, Sungsu Lim, Ajin George Joseph, Y angchen Pan, Adam White, and Martha White. Greedy actor-critic: A new conditional cross-entropy method for policy im- provement. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  18. [18]

    Offline reinforcement learn- ing with tsallis regularization.Transactions on Machine Learning Research, 2024

    Lingwei Zhu, Matthew Schlegel, Han Wang, and Martha White. Offline reinforcement learn- ing with tsallis regularization.Transactions on Machine Learning Research, 2024

    2024

  19. [19]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InProceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1889–1897, 2015

    2015

  20. [20]

    Proximal policy optimization algorithms.arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  21. [21]

    Lillicrap, Jonathan J

    Timothy P . Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. In4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016

    2016

  22. [22]

    Al-Sakkari, Ahmed Ragab, Mohamed Ali, Hanane Dagdougui, and Daria C

    Eslam G. Al-Sakkari, Ahmed Ragab, Mohamed Ali, Hanane Dagdougui, and Daria C. Boffito. Simulate intelligently: Causal incremental reinforcement learning for streamlined industrial chemical process design optimization.Journal of Environmental Chemical Engineering, 13(6):120167, 2025. ISSN 2213-3437. doi:https://doi.org/10.1016/j.jece.2025. 120167

    work page doi:10.1016/j.jece.2025 2025

  23. [23]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning.Machine Learning, page 229–256, May 1992

    1992

  24. [24]

    Rupam Mahmood, and Martha White

    Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences.Journal of Machine Learning Research, 23(253):1–79, 2022

    2022

  25. [25]

    Mirror descent and nonlinear projected subgradient methods for convex optimization.Operations Research Letters, 31(3):167–175, 2003

    Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization.Operations Research Letters, 31(3):167–175, 2003

    2003

  26. [26]

    Investigating the utility of mirror descent in off-policy actor-critic

    Samuel Neumann, Jiamin He, Adam White, and Martha White. Investigating the utility of mirror descent in off-policy actor-critic. InReinforcement Learning Journal, 2025

    2025

  27. [27]

    Leverage the average: an analysis of regularization in rl

    Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of regularization in rl. InAdvances in Neural Information Processing Systems 33, pages 1–12, 2020

    2020

  28. [28]

    Mirror descent policy optimization

    Manan Tomar, Lior Shani, Y onathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. InInternational Conference on Learning Representations, 2022

    2022

  29. [29]

    Machado, Pablo Samuel Castro, and Nicolas Le Roux

    Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C. Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. InProceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 8619–8649, 2022

    2022

  30. [30]

    Approximately optimal approximate reinforcement learn- ing

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learn- ing. In19th International Conference on Machine Learning (ICML), pages 267–274, 2002

    2002

  31. [31]

    Revisiting mixture policies in entropy-regularized actor-critic

    Jiamin He, Samuel Neumann, Jincheng Mei, Adam White, and Martha White. Revisiting mixture policies in entropy-regularized actor-critic. InSecond Workshop on Aligning Rein- forcement Learning Experimentalists and Theorists of NeurIPS 2025, 2026

    2025

  32. [32]

    Improving stochastic policy gradi- ents in continuous control with deep reinforcement learning using the beta distribution

    Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradi- ents in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning, pages 834–843. PMLR, 2017

    2017

  33. [33]

    Student-t policy in reinforcement learning to acquire global optimum of robot control.Applied Intelligence, 49(12):4335–4347, 2019

    Taisuke Kobayashi. Student-t policy in reinforcement learning to acquire global optimum of robot control.Applied Intelligence, 49(12):4335–4347, 2019

    2019

  34. [34]

    q-exponential policy optimization

    Lingwei Zhu, Haseeb Shah, Han Wang, Yukie Nagai, and Martha White. q-exponential policy optimization. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  35. [35]

    Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Long Y ang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Y ang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Diffusion-based reinforcement learning via q-weighted variational policy opti- mization.Advances in Neural Information Processing Systems, 37:53945–53968, 2024

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Y e Shi. Diffusion-based reinforcement learning via q-weighted variational policy opti- mization.Advances in Neural Information Processing Systems, 37:53945–53968, 2024

    2024

  38. [38]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  39. [39]

    Sampling from energy- based policies using diffusion.Reinforcement Learning Journal, 6:2291–2307, 2025

    Vineet Jain, Tara Akhound-Sadegh, and Siamak Ravanbakhsh. Sampling from energy- based policies using diffusion.Reinforcement Learning Journal, 6:2291–2307, 2025

    2025

  40. [40]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. InProceedings of the 39th International Conference on Machine Learning, pages 16828–16847, 2022

    2022

  41. [41]

    Addressing the plasticity- stability dilemma in reinforcement learning.arXiv preprint arXiv:2512.01034, 2025

    Mansi Maheshwari, John C Raisbeck, and Bruno Castro da Silva. Addressing the plasticity- stability dilemma in reinforcement learning.arXiv preprint arXiv:2512.01034, 2025

    arXiv 2025

  42. [42]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D’Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. InDeep reinforcement learning workshop neurIPS 2022, 2022

    2022

  43. [43]

    Mad-td: Model-augmented data stabilizes high update ratio rl

    Claas Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand, and Igor Gilitschen- ski. Mad-td: Model-augmented data stabilizes high update ratio rl. InInternational Confer- ence on Learning Representations, volume 2025, pages 71619–71649, 2025

    2025

  44. [44]

    Stochastic approximation with two time scales.Systems & Control Letters, 29(5):291–294, 1997

    Vivek S Borkar. Stochastic approximation with two time scales.Systems & Control Letters, 29(5):291–294, 1997

    1997

  45. [45]

    Actor-critic algorithms.Advances in neural information processing systems, 12, 1999

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms.Advances in neural information processing systems, 12, 1999

    1999

  46. [46]

    Sutton, Mohammad Ghavamzadeh, and Mark Lee

    Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor–critic algorithms.Automatica, 45(11):2471–2482, 2009

    2009

  47. [47]

    Likelihood ratio gradient estimation for stochastic systems.Communications of the ACM, 33(10):75–84, 1990

    Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems.Communications of the ACM, 33(10):75–84, 1990

    1990

  48. [48]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Pith/arXiv arXiv 2013

  49. [49]

    Categorical reparameterization with gumbel- softmax.arXiv preprint arXiv:1611.01144, 2016

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax.arXiv preprint arXiv:1611.01144, 2016

    Pith/arXiv arXiv 2016

  50. [50]

    Variance reduction properties of the reparameterization trick

    Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction properties of the reparameterization trick. InThe 22nd international conference on artificial intelligence and statistics, pages 2711–2720. PMLR, 2019

    2019

  51. [51]

    Pipps: Flexible model-based policy search robust to the curse of chaos

    Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. InInternational Conference on Machine Learning, pages 4065–4074. PMLR, 2018

    2018

  52. [52]

    Pid controllers: theory, design, and tuning.The international society of measurement and control, 1995

    Karl J Astrom. Pid controllers: theory, design, and tuning.The international society of measurement and control, 1995

    1995

  53. [53]

    Fat-to-thin policy optimization: Offline rl with sparse policies

    Lingwei Zhu, Han Wang, and Yukie Nagai. Fat-to-thin policy optimization: Offline rl with sparse policies. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  54. [54]

    Y ana Suchikova, Natalia Tsybuliak, Jaime A Teixeira da Silva, and Serhii Nazarovets. Gaidet (generative ai delegation taxonomy): A taxonomy for humans to delegate tasks to generative artificial intelligence in scientific research and publishing.Accountability in Research, 33(3): 2544331, 2026. 12 of 38 Supplementary Materials Contents 1 Reward Surfaces ....

    2026

  55. [55]

    Clipped Gaussian Sampling.In this approach, the raw meanµraw is passed through atanhnonlinearity and scaled to fit within the action bounds: µ=a min + amax−a min 2 (tanh(µraw) + 1)(8) a=µ+σ·ϵ, ϵ∼N(0,1)(9) ascaled =clip(a,a min,a max)(10)

  56. [56]

    Squashed Gaussian Sampling.An alternative approach is to sample from the Gaussian and apply a smooth tanhsquashing function: ˜a=µ+σ·ϵ, ϵ∼N(0,1), a= tanh(˜a)(11) followed by rescaling: ascaled =a min + amax−a min 2 (a+ 1)(12) The squashed Gaussian applies tanh directly to the sampled action, which alters the distribution’s support and requires a log-determ...