Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Wasserstein Policy Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Wasserstein Policy Optimization claims a single closed-form update that uses action-value gradients yet works for arbitrary stochastic policies without reparameterization.

desk verdict A genuinely new parametric update for stochastic control with a clean Gaussian analysis, but the 'completely general' claim is undercut by a dropped boundary term in Eq. 5 and a missing ascent proof for non-Gaussians. read the letter →

arxiv 2505.00663 v1 pith:P2XFTJ5K submitted 2025-05-01 cs.LG cs.AI

David Pfau , Ian Davies , Diana Borsa , Joao G. M. Araujo , Brendan Tracey , Hado van Hasselt This is my paper · ORCID
classification cs.LGcs.AI
keywords Wassersteingradientflowpolicyoptimizationactor-criticcontinuouscontroldeterministicreparameterizationtrickstochasticpolicieshigh-dimensionalactionspaces
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give reinforcement learning in continuous action spaces a policy update that is a true gradient step in the space of probability distributions over actions, not just in parameter space. It derives Wasserstein Policy Optimization (WPO) as the parametric projection of the Wasserstein gradient flow that maximizes expected return, and arrives at a closed-form formula: the natural-gradient-rescaled expectation of $\nabla_\theta \nabla_a \log \pi$ times the action-value gradient $\nabla_a Q^\pi(s,a)$. If this derivation is right, one update rule combines the two families that were previously separate: it exploits action-value gradients (as deterministic policy gradient does) while still updating any stochastic policy by plain score-function sampling, with no reparameterization trick. This matters because deterministic methods explore poorly and reparameterized stochastic methods exclude distributions like mixtures of Gaussians; WPO claims to avoid both limitations. The paper supports this claim with a Gaussian-case equivalence result, a mixture-of-Gaussians illustration, and experiments on continuous-control benchmarks and a tokamak magnetic-control task.

What carries the argument

The load-bearing object is the Wasserstein gradient-flow PDE $\partial \pi / \partial t = -\nabla_a \cdot (\pi \nabla_a Q^\pi)$ (with sign chosen for ascent), the steepest descent of expected return under the 2-Wasserstein metric. The paper's step is to project this nonparametric flow onto a parametric family by minimizing the KL divergence between the updated and the target policy; expanding that KL to second order gives the Fisher matrix $F_{\theta\theta}$ and the cross term $F_{t\theta} = \mathbb{E}_{a\sim\pi}[\nabla_\theta \nabla_a \log \pi_\theta(a|s) \, \nabla_a Q^\pi(s,a)]$, obtained by integration by parts. The resulting closed-form update $\theta \leftarrow \theta + F_{\theta\theta}^{-1} F_{t\theta}$ is what carries the argument. In practice the paper specializes to diagonal Gaussian policies, rescales mean and variance gradients by $\sigma_i^2$ and $\tfrac{1}{2}\sigma_i^2$ to approximate the Fisher inverse, and adds a KL penalty against the previous policy to prevent premature collapse; it also notes that the c-Wasserstein extension replaces $\nabla_a Q$ by a squashed version $\nabla c^*(\nabla_a Q)$.

What would settle it

Take a one-state MDP with action-value $Q(a) = -a^4/100 + a^2$ and a two-component Gaussian-mixture policy initialized with means at $\pm 1$ and large variances, the setting of the paper's Figure 3; compute the exact expected WPO update with the full Fisher matrix and no KL penalty, and check whether the return $J[\pi]$ increases for a sufficiently small step size. A step that decreases $J$, or fails to increase it as the step size tends to zero, would contradict the claim that WPO is an ascent direction.

Watch

Extended reading notes

Core claim

The central claim is that WPO is a simple and completely general closed-form policy update that is a parametric approximation to the Wasserstein gradient flow of the expected-return functional. For any policy $\pi$, the update is $\theta \leftarrow \theta + F_{\theta\theta}^{-1} \mathbb{E}_{a\sim\pi}[\nabla_\theta \nabla_a \log \pi_\theta(a|s) \, \nabla_a Q^\pi(s,a)]$, where $F_{\theta\theta}$ is the Fisher information matrix of the policy; the mixed score-function derivative lets the algorithm carry the action-value gradient through the policy without reparameterizing the sampling distribution. The functional derivative $\delta J/\delta \pi = (1/(1-\gamma)) Q^\pi d^\pi$ makes $Q^\pi$ the driving field, so the update inherits DPG's use of $\nabla_a Q^\pi$, while the expectation over $a \sim \pi$ keeps it a stochastic-policy method. For Gaussian policies the expected WPO update equals both the classic policy gradient and the reparameterized DPG update, so the paper argues that the differences surface only for non-Gaussian distributions such as mixtures of Gaussians, where WPO is applicable and behaves distinctly. Empirically, WPO is competitive with DDPG, SAC, and MPO across the DeepMind Control Suite, learns faster on replicated high-dimensional control tasks (up to 105 action dimensions), and matches MPO on a simulated tokamak magnetic-control task while driving the policy variance down.

Load-bearing premise

The central assumption is that replacing the true Wasserstein steepest-descent direction with its best local KL-divergence approximation inside the parameterized policy family still points uphill in expected return; the paper gives no proof of this for non-Gaussian policies.

Editorial extensions

If this is right

  • WPO gives a single update rule that reduces to the standard policy gradient for Gaussian policies while matching the reparameterized DPG update, so the method inherits exploration from stochastic sampling and the variance reduction of action-value gradients.
  • Because no reparameterization is needed, any policy with a computable mixed score-function gradient—including mixture-of-Gaussians or exponential distributions—can be trained by WPO, opening a class of policies that stochastic DPG methods cannot handle exactly.
  • On high-dimensional continuous control (replicated Humanoid tasks with 21, 65, and 105 action dimensions), the paper reports that WPO learns faster than DDPG, SAC, and MPO, with the advantage growing as the action dimension grows.
  • On the tokamak magnetic-control task, WPO matches MPO's reward while its policy variance shrinks toward deterministic over training, which the paper interprets as the expected behavior for a fully observed environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the KL projection is a faithful surrogate for the Wasserstein flow, WPO's dynamics are governed by the 2-Wasserstein geometry rather than by the Fisher metric; a testable consequence is that WPO should be more robust to reparameterizations of the policy than natural policy gradient, a contrast the paper's exponential-policy example begins to illustrate.
  • The paper's observation that sampled WPO updates have zero variance for locally linear $Q$ suggests that its variance reduction should become more pronounced as the action dimension grows; one can test this directly by measuring gradient variance of WPO versus classic policy gradient on the replicated high-dimensional tasks.
  • The c-Wasserstein squashing variant (cube root applied to $\nabla_a Q$) opens a family of updates, each a valid descent direction for a different cost function; comparing squashing functions on tasks with rapidly changing $Q$ could identify which geometric penalization is best for stability.
  • Because WPO is presented as a genuine gradient method in distribution space, it may combine naturally with entropy or KL bonuses as regularizers rather than as parts of the objective, potentially yielding a cleaner exploration-exploitation trade-off than a tuned entropy coefficient.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes Wasserstein Policy Optimization (WPO), an actor-critic algorithm whose update is obtained by projecting the Wasserstein gradient flow of the return functional onto a parametric policy family. The central object is the closed-form update Δθ = F^{-1} E_π[∇θ∇a log π ∇a Q] (Eq. 6). The paper shows that for Gaussian policies this update coincides with the standard policy gradient and with reparameterized DPG/SVG-style updates, analyzes a mixture-of-Gaussians example, and then introduces a practical diagonal-Gaussian implementation with variance rescaling and KL regularization. Experiments cover DeepMind Control Suite, replicated high-dimensional tasks, and a TCV tokamak magnetic-control task, reporting competitive or faster learning relative to MPO, SAC, and DDPG.

Significance. WPO is an interesting and potentially useful update: it gives a stochastic-policy analogue of deterministic policy gradients without the reparameterization trick, and the Gaussian analysis is clean. The paper contributes a broad empirical comparison, an open-source Acme implementation, and an exact derivation for full-support Gaussian policies. However, the headline claim of a 'completely general' update is currently stronger than what is proved: the derivation has a boundary-term flaw for constrained-support policies, and the non-Gaussian projection step is not shown to preserve ascent. If the generality claim were restricted or proved, the paper would be a solid contribution; as written, the theoretical part needs revision.

major comments (3)
  1. [§2.3 / App. A.2, Eq. (26)] The integration by parts leading to Eq. (5) drops the boundary term under the assumption that ∇θπθ(a|s) vanishes at infinity. This fails for any policy with constrained support. A concrete counterexample is the exponential policy π(a)=β^{-1}e^{-a/β} on [0,∞) used in §4.1: ∇βπ(0)=-β^{-2}, so the corrected identity is F_tβ=(Eπ[∇aQ]-∇aQ(0))/β², not Eπ[∇aQ]/β², and the update shown in that section is not the WPO update for this family. The 'completely general' claim in the abstract and §2.3 should be restricted to policies whose support is the full action space with appropriate decay, or the update should be modified to include the boundary term.
  2. [§2.3] The projection step from the nonparametric Wasserstein flow to the parametric update minimizes a local KL/Fisher quadratic, but no argument shows that the resulting discrete step increases J[πθ] for non-Gaussian policies. The continuous-time flow is an ascent in the Wasserstein metric on densities; ascent does not automatically survive projection in the KL/Fisher metric, and the paper provides no bound on or sign analysis of ⟨∇θJ, F^{-1}F_tθ⟩ outside the Gaussian case. The mixture-of-Gaussians example in §4.2 is a single favorable numerical instance, so the 'completely general' update is not established by the evidence presented.
  3. [§4.2 / §5] The non-Gaussian evidence is weakened by the fact that the mixture-of-Gaussians experiment uses the heuristic rescaling of gradients by σ_i² rather than the FIM of the mixture, and the practical algorithm in §5 additionally restricts to diagonal Gaussian policies and replaces the full Fisher by a variance rescaling. The text is transparent about these choices, but the experimental results therefore do not validate Eq. (6) for arbitrary stochastic policies; the abstract and conclusion should align the generality claim with what is actually implemented and tested.
minor comments (5)
  1. [Fig. 3 caption] The caption refers to the 'left plot' twice and is inconsistent about which panel shows the policy-gradient evolution versus the WPO evolution; please relabel or re-describe the panels.
  2. [Fig. 6 caption] The caption says 'fusion task discussed in Section 6.2', but the fusion task is presented in Section 6.3.
  3. [§4.1] The claim that the Gaussian equivalences extend 'straightforwardly' to multivariate normals would benefit from an appendix derivation, because the covariance parameterization introduces off-diagonal terms.
  4. [Footnote 1] The footnote states that the public Acme implementation is not the version used for the experiments; please state clearly in the main text whether the exact experimental configuration will be released.
  5. [App. A.2, Eq. (26)] The integration-by-parts assumption should be stated as a decay condition on the full boundary term, e.g., ∇θlogπ(a|s) π(a|s) ∇aQ(s,a) → 0, rather than only on ∇θπ(a|s); this is the condition actually needed for Eq. (30) to hold.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WPO's update is derived from an independent variational argument and validated against external benchmarks.

full rationale

The central derivation is self-contained. Starting from the functional derivative of J[pi] with respect to the policy, Eq. 4, the paper constructs the Wasserstein gradient flow (Eq. 1), then projects it onto a parametric family by minimizing a local KL/Fisher quadratic form (Sec. 2.3). The resulting closed-form update, Eq. 6, is not an input or a renamed fit: no parameter of the update is fitted to the data used to test it, and the action-value gradient E[grad_theta grad_a log pi grad_a Q] follows from integration by parts rather than from assuming the conclusion. The Gaussian-case equivalence with DPG and classic policy gradient is an independently derived identity, not a definitional restatement, and the mixture-of-Gaussians and Control Suite experiments test the method against external baselines rather than assuming its success. Self-citations (e.g., Tracey et al. 2024 for the fusion task, Degrave et al. 2022 for TCV, Hoffman et al. 2020 for Acme) concern benchmark settings, implementations, and prior experimental conventions; they do not carry the mathematical derivation. The stated boundary assumption in App. A.2 that grad_theta pi vanishes at infinity is an explicit technical assumption, and the exponential-policy counterexample in the skeptic note is a correctness or generality concern about that assumption, not a circularity: the paper does not define the conclusion in terms of the assumption. No load-bearing step reduces by construction to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central derivation contains no fitted constants. The practical algorithm introduces hand-chosen heuristics: diagonal variance rescaling, KL penalty weights, and a squashing function for the fusion task. The projection is via KL, not the Wasserstein metric, which is the main assumption that could be challenged.

free parameters (3)
  • WPO KL penalty weights alpha_mu and alpha_Sigma = alpha_mu = log(2), alpha_Sigma = 10000 (Table 2)
    Chosen by hand for Control Suite performance; the variance penalty is needed to prevent collapse.
  • Variance rescaling in diagonal Fisher approximation = grad_bar_mu = sigma^2 grad_mu, grad_bar_sigma = 1/2 sigma^2 grad_sigma
    Heuristic rescaling replacing the natural gradient, justified in Sec. 5 to prevent blowup; not derived from the Wasserstein flow.
  • Cube-root squashing (fusion task) = grad c*(x) = x^{1/3}
    Chosen for numerical stability; based on c-Wasserstein theory but the specific convex function is ad hoc.
assumptions (5)
  • domain assumption The functional derivative of the expected return with respect to a continuous policy is delta J / delta pi = (1/(1-gamma)) Q^pi(s,a) d^pi(s).
    Derived in Appendix A.1 under standard MDP smoothness; used as the gradient field for the Wasserstein flow (Eq. 4).
  • standard math The PDE d(pi)/dt = -nabla_a dot (pi nabla_a Q) is the steepest ascent of J in the 2-Wasserstein metric.
    Standard optimal transport result (Ambrosio et al., 2008), cited in Sec. 2.2; this is the theoretical foundation.
  • ad hoc to paper KL divergence is locally approximated by the Fisher quadratic form, and minimizing it yields the correct parametric projection of the flow.
    Sec. 2.3 uses KL, not the Wasserstein distance, for the projection; this choice is motivated by tractability and is not the metric of the flow.
  • standard math Integration by parts boundary term vanishes: nabla_theta pi_theta(a|s) -> 0 as |a| -> infinity.
    Assumed in Appendix A.2, Eq. 26; holds for Gaussians and most parametric policies considered, but not for heavy tails.
  • domain assumption Per-state updates can ignore the discounted occupancy d^pi(s) and the (1-gamma)^{-1} factor.
    Sec. 2.2 states these terms appear as sampling frequency; the per-state approximation is standard in actor-critic methods but is a simplification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Wasserstein Policy Optimization." pith.science (2026). https://pith.science/paper/P2XFTJ5K

@misc{pith2026250500663,
  author       = {Pith},
  title        = {Pith review of: Wasserstein Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P2XFTJ5K}},
  note         = {Machine review of arXiv:2505.00663}
}
read the original abstract

We introduce Wasserstein Policy Optimization (WPO), an actor-critic algorithm for reinforcement learning in continuous action spaces. WPO can be derived as an approximation to Wasserstein gradient flow over the space of all policies projected into a finite-dimensional parameter space (e.g., the weights of a neural network), leading to a simple and completely general closed-form update. The resulting algorithm combines many properties of deterministic and classic policy gradient methods. Like deterministic policy gradients, it exploits knowledge of the gradient of the action-value function with respect to the action. Like classic policy gradients, it can be applied to stochastic policies with arbitrary distributions over actions -- without using the reparameterization trick. We show results on the DeepMind Control Suite and a magnetic confinement fusion task which compare favorably with state-of-the-art continuous control methods.

Figures

Figures reproduced from arXiv: 2505.00663 by the authors.

Figure 1
Figure 1. Conceptual illustration of how WPO combines elements of stochastic and deterministic policy gradient methods. Left: “classic” policy gradient. Samples are taken from a stochastic policy. Each sample contributes a scalar Q π (s, a) factor to the gradient. Middle: deterministic policy gradient (DPG). A deterministic action is chosen and the policy gradient depends on the gradient of Q π (s, a). Right: Wasserstein poli… view at source ↗
Figure 2
Figure 2. Concrete WPO updates for a single-variate normal policy for two different action-value functions. In the left and middle plots we consider Q(a) = −a 2 /2, with an obvious optimum at a = 0, and a policy with µ = σ = 1. The left plot shows the gradient on the mean at several sampled actions. These are averaged to produce the update, moving the mean towards the optimal action. The expected WPO update, as shown in the m… view at source ↗
Figure 3
Figure 3. Concrete WPO learning for a one dimensional mixture of Gaussians policy for the non-concave action-value function Q(a) = − 1 100 a 4 + a 2 . The left plot shows the action-value function and mixture of Gaussians policy. In the middle plot we show the evolution of the policy under a standard policy gradient update, both with samples from the policy and the change in the means of each mixture component. On the left we… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Results from selected DeepMind Control Suite tasks. Full results are in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Plots of reward from various agents on combined Humanoid Stand environments. Left to right: 1, 3 and 5 replicated environments (21, 65 and 105 action dimension). Solid line denotes the mean and the shaded region highlights the minimum and maximum over 5 seeds. As the n…
Figure 6
Figure 6. Figure 6: Reward and policy average standard deviation evolution throughout training on the fusion task discussed in Section 6.2 the environments. In the singleton case, all methods are qualitatively similar except SAC which converges to a lower reward than other learning algori…
Figure 7
Figure 7. Figure 7: Results on all DeepMind Control Suite tasks for WPO, MPO, DDPG and SAC. The bold lines denote average returns over 10 seeds on evaluation episodes. The shaded region spans minimum and maximum returns. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Schematic of the TCV tokamak, including cutout showing 2-D profile of plasma contour B.2. Combined Tasks In order to combine the rewards across component environments in the combined tasks we apply a SmoothMin operation. This is the application of a SmoothMax operation…
Figure 9
Figure 9. Figure 9: The reference control points and example flux field for the shape 70166 task. The last closed flux surface target points are shown in blue on the left, with the limit point location in cyan. An example magnetic flux field matching these control points in shown on the l…
Figure 10
Figure 10. Figure 10: Return and policy average standard-deviation evolution throughout training on the shape 70166 task from (Tracey et al., 2024) C. Additional Results and Ablations C.1. Complete DeepMind Control Suite Results In [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Results on all DeepMind Control Suite tasks for MPO with different hyperparameters: optimal settings tuned for Control Suite, settings matching WPO, and settings matching WPO with variance rescaling. The performance of MPO is not significantly impacted by these choice…
Figure 12
Figure 12. Figure 12: Results on all DeepMind Control Suite tasks for WPO with different activation functions (ELU vs. SiLU) and with and without applying a cube root elementwise to the action-value gradients. The bold lines denote average returns over 3 seeds on evaluation episodes. The s…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 30 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M

    Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018

    work page 2018

  3. [3]

    A., Ren, H., Ammar, H

    Abdullah, M. A., Ren, H., Ammar, H. B., Milenkovic, V., Luo, R., Zhang, M., and Wang, J. Wasserstein robust reinforcement learning, 2019. URL https://arxiv.org/abs/1907.13196

    arXiv 2019

  4. [4]

    M., Lee, J

    Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22 0 (98): 0 1--76, 2021

    2021

  5. [5]

    Gradient flows: in metric spaces and in the space of probability measures

    Ambrosio, L., Gigli, N., and Savar \'e , G. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008

    2008

  6. [6]

    W., Budden, D., Dabney, W., Horgan, D., Tb, D., Muldal, A., Heess, N., and Lillicrap, T

    Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Tb, D., Muldal, A., Heess, N., and Lillicrap, T. Distributed distributional deterministic policy gradients. International Conference on Learning Representations, 2018

    work page 2018

  7. [7]

    G., Sutton, R

    Barto, A. G., Sutton, R. S., and Anderson, C. W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, 0 (5): 0 834--846, 1983

    work page 1983

  8. [8]

    G., Dabney, W., and Munos, R

    Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp.\ 449--458. PMLR, 2017

    work page 2017

Show all 54 references
  1. [9]

    and Brenier, Y

    Benamou, J.-D. and Brenier, Y. A computational fluid mechanics solution to the monge-kantorovich mass transfer problem. Numerische Mathematik, 84 0 (3): 0 375--393, 2000

    2000

  2. [10]

    Development of free-boundary equilibrium and transport solvers for simulation and real-time interpretation of tokamak experiments

    Carpanese, F. Development of free-boundary equilibrium and transport solvers for simulation and real-time interpretation of tokamak experiments. Technical report, EPFL, 2021

    2021

  3. [11]

    S., Kastner, T., Panangaden, P., and Rowland, M

    Castro, P. S., Kastner, T., Panangaden, P., and Rowland, M. Mico: Improved representations via sampling-based state similarity for markov decision processes, 2022. URL https://arxiv.org/abs/2106.08229

    2022 arXiv

  4. [12]

    T., Rubanova, Y., Bettencourt, J., and Duvenaud, D

    Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  5. [13]

    Fast and accurate deep network learning by exponential linear units (elus)

    Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arxiv 2015. International Conference on Learning Representations, 2016

    2015

  6. [14]

    Magnetic control of tokamak plasmas through deep reinforcement learning

    Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602 0 (7897): 0 414--419, 2022

    2022

  7. [15]

    Experimental research on the TCV tokamak

    Duval, B., Abdolmaleki, A., Agostini, M., Ajay, C., Alberti, S., Alessi, E., Anastasiou, G., Andrèbe, Y., Apruzzese, G., Auriemma, F., et al. Experimental research on the TCV tokamak. Nuclear Fusion, 64 0 (11): 0 112023, oct 2024. doi:10.1088/1741-4326/ad8361. URL https://dx.d...

    2024 doi

  8. [16]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

    Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107: 0 3--11, 2018

    2018

  9. [17]

    and Castro, P

    Farebrother, J. and Castro, P. S. Cale: Continuous arcade learning environment. arXiv preprint arXiv:2410.23810, 2024

    2024 arXiv

  10. [18]

    Metrics for finite markov decision processes

    Ferns, N., Panangaden, P., and Precup, D. Metrics for finite markov decision processes. In Uncertainty in Artificial Intelligence, 2004

    2004

  11. [19]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  12. [20]

    H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N

    Haarnoja, T., Moran, B., Lever, G., Huang, S. H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N. Y., Hafner, R., et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Science Robotics, 9 0 (89): 0 eadi8022, 2024

    2024

  13. [21]

    Wasserstein unsupervised reinforcement learning, 2021

    He, S., Jiang, Y., Zhang, H., Shao, J., and Ji, X. Wasserstein unsupervised reinforcement learning, 2021. URL https://arxiv.org/abs/2110.07940

    2021 arXiv

  14. [22]

    Learning continuous control policies by stochastic value gradients

    Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. Learning continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015

    2015

  15. [23]

    W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Sta \'n czyk, P., Ramos, S., Raichuk, A., Vincent, D., et al

    Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Sta \'n czyk, P., Ramos, S., Raichuk, A., Vincent, D., et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020

    2006 arXiv

  16. [24]

    Howard, R. A. Dynamic P rogramming and M arkov P rocesses . 1960

    1960

  17. [25]

    Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022

    2023

  18. [26]

    Categorical reparameterization with G umbel-softmax

    Jang, E., Gu, S., and Poole, B. Categorical reparameterization with G umbel-softmax. ICLR, 2017

    2017

  19. [27]

    Kakade, S. M. A natural policy gradient. Advances in neural information processing systems, 14, 2001

    2001

  20. [28]

    M., and Restelli, M

    Likmeta, A., Sacco, M., Metelli, A. M., and Restelli, M. Wasserstein actor-critic: Directed exploration via optimism for continuous-actions control, 2023. URL https://arxiv.org/abs/2303.02378

    2023 arXiv

  21. [29]

    Continuous control with deep reinforcement learning

    Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  22. [30]

    J., Mnih, A., and Teh, Y

    Maddison, C. J., Mnih, A., and Teh, Y. W. The C oncrete D istribution: A C ontinuous R elaxation of D iscrete R andom V ariables. ICLR, 2017

    2017

  23. [31]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with K ronecker-factored approximate curvature. In International Conference on Machine Learning, pp.\ 2408--2417. PMLR, 2015

    2015

  24. [32]

    M., Likmeta, A., and Restelli, M

    Metelli, A. M., Likmeta, A., and Restelli, M. Propagating uncertainty in reinforcement learning via wasserstein barycenters. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  25. [33]

    Efficient W asserstein natural gradients for reinforcement learning

    Moskovitz, T., Arbel, M., Huszar, F., and Gretton, A. Efficient W asserstein natural gradients for reinforcement learning. arXiv preprint arXiv:2010.05380, 2020

    2010 arXiv

  26. [34]

    Wasserstein quantum M onte C arlo: a novel approach for solving the quantum many-body schr \"o dinger equation

    Neklyudov, K., Nys, J., Thiede, L., Carrasquilla, J., Liu, Q., Welling, M., and Makhzani, A. Wasserstein quantum M onte C arlo: a novel approach for solving the quantum many-body schr \"o dinger equation. Advances in Neural Information Processing Systems, 36, 2023

    2023

  27. [35]

    Learning to score behaviors for guided policy optimization

    Pacchiano, A., Parker-Holder, J., Tang, Y., Choromanski, K., Choromanska, A., and Jordan, M. Learning to score behaviors for guided policy optimization. In International Conference on Machine Learning, pp.\ 7445--7454. PMLR, 2020

    2020

  28. [36]

    and Bengio, Y

    Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013

    2013 arXiv

  29. [37]

    Prokhorov, D. V. and Wunsch, D. C. Adaptive critic designs. IEEE transactions on Neural Networks, 8 0 (5): 0 997--1007, 1997

    1997

  30. [38]

    Richemond, P. H. and Maginnis, B. Diffusing policies : Towards W asserstein policy gradient flows, 2018. URL https://openreview.net/forum?id=rk3mjYRp-

    2018

  31. [39]

    Trust region policy optimization

    Schulman, J., Levine, S., Moritz, P., Jordan, M., and Abbeel, P. Trust region policy optimization. International Conference on Machine Learning, 2015 a

    2015

  32. [40]

    High-dimensional continuous control using generalized advantage estimation

    Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015 b

    2015 arXiv

  33. [41]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  34. [42]

    Deterministic policy gradient algorithms

    Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp.\ 387--395. PMLR, 2014

    2014

  35. [43]

    F., Abdolmaleki, A., Springenberg, J

    Song, H. F., Abdolmaleki, A., Springenberg, J. T., Clark, A., Soyer, H., Rae, J. W., Noury, S., Ahuja, A., Liu, S., Tirumala, D., et al. V-MPO : O n-policy maximum a posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv:1909.12238, 2019

    1909 arXiv

  36. [44]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  37. [45]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999

    1999

  38. [46]

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  39. [47]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  40. [48]

    D., Michi, A., Chervonyi, Y., Davies, I., Paduraru, C., Lazic, N., Felici, F., Ewalds, T., Donner, C., Galperti, C., et al

    Tracey, B. D., Michi, A., Chervonyi, Y., Davies, I., Paduraru, C., Lazic, N., Felici, F., Ewalds, T., Donner, C., Galperti, C., et al. Towards practical reinforcement learning for tokamak magnetic control. Fusion Engineering and Design, 200: 0 114161, 2024

    2024

  41. [49]

    dm\_control: Software and tasks for continuous control

    Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020

    2020

  42. [50]

    Double q-learning

    van Hasselt , H. Double q-learning. In Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., and Culotta, A. (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010

    2010

  43. [51]

    and Wiering, M

    van Hasselt , H. and Wiering, M. A. Reinforcement learning in continuous action spaces. In 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp.\ 272--279. IEEE, 2007

    2007

  44. [52]

    Beyond regression: N ew tools for prediction and analysis in the behavioral sciences

    Werbos, P. Beyond regression: N ew tools for prediction and analysis in the behavioral sciences. PhD thesis, Committee on Applied Mathematics, Harvard University, Cambridge, MA, 1974

    1974

  45. [53]

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  46. [54]

    Policy optimization as W asserstein gradient flows

    Zhang, R., Chen, C., Li, C., and Carin, L. Policy optimization as W asserstein gradient flows. In International Conference on Machine Learning, pp.\ 5737--5746. PMLR, 2018

    2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.