Pith. sign in

REVIEW 5 major objections 5 minor 18 references

Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WAVE adds an adaptively weighted Wasserstein penalty between successive Q-value distributions to the critic loss, and the paper claims this makes the regularized critic's mean squared error fall at O(1/k) while the regularized Bellman…

desk verdict A genuinely new algorithmic idea undermined by overstated and flawed theoretical claims; the empirical observation is plausible but under-reported. read the letter →

arxiv 2501.10605 v2 pith:LMJD7D2S submitted 2025-01-17 cs.LG cs.SYeess.SYstat.ML

Ali Baheri , Zahra Shahrooei , Chirayu Salgarkar This is my paper · ORCID
classification cs.LGcs.SYeess.SYstat.ML
keywords Wassersteindistanceoptimaltransportactor-criticmethodsreinforcementlearningSinkhornapproximationadaptiveregularizationconvergenceanalysiscontinuouscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that the instability of actor-critic reinforcement learning can be reduced by adding a Wasserstein penalty between successive Q-value distributions to the critic's loss, with the penalty's weight adapted to the agent's recent rewards. It claims that this Wasserstein Adaptive Value Estimation (WAVE) makes the critic's mean squared error decrease at an O(1/k) rate, the critic parameters converge at O(1/sqrt(k)), and the regularized Bellman operator contract faster than the unregularized one. The point of the claim is practical: deep actor-critic agents often fail because value estimates fluctuate, and a cheap, differentiable Sinkhorn approximation of the Wasserstein distance can be inserted into the critic update to enforce temporal consistency without changing the actor.

What carries the argument

The load-bearing object is the Sinkhorn distance $W_\varepsilon(\hat Q_k, \hat Q_{k-1})$, the entropically regularized optimal-transport plan between consecutive empirical distributions of critic outputs. It is differentiable in the critic parameters, so its gradient can be added directly to the TD gradient, and it tracks a geometry-aware notion of how much the Q-value distribution moved between updates. The other carrying element is the adaptive coefficient $\lambda_k$ defined from the moving average of episode rewards, and, in the convergence proof, the strong convexity of $L_{\mathrm{reg}}$ with parameter $m$ together with bounded gradients, which converts the gradient descent recursion into the $1/k$ bound.

What would settle it

Compute the smallest eigenvalue of the Hessian of the regularized critic loss on a three-layer critic at the states a run of WAVE actually visits; if that eigenvalue is zero or negative for any visited point, strong convexity is violated and the O(1/k) proof does not apply. A more direct check is to run WAVE with the Wasserstein term removed and compare the empirical squared parameter error against the predicted C/k curve: if the regularized and unregularized errors have the same rate and nearly the same constant, the claimed stabilizing mechanism is not carrying the result.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the regularized critic loss $L_{\mathrm{reg}}(\theta^Q) = L_{\mathrm{TD}}(\theta^Q) + \lambda_k W_\varepsilon(\hat Q_k, \hat Q_{k-1})$ -- the temporal-difference error plus an adaptive Sinkhorn-approximated Wasserstein distance between the empirical Q-value distributions of consecutive iterations -- yields provably stable critic updates. Theorem 1 shows that when $L_{\mathrm{reg}}$ is strongly convex, the expected squared parameter error obeys $\mathbb{E}[\|\theta_k - \theta^*\|^2] \le C/k$. Theorem 2 argues that the penalty lowers gradient variance by opposing TD gradients, and Theorem 3 claims the regularized Bellman operator is a contraction with factor $\gamma(1 - c\lambda)$, strictly smaller than $\gamma$. The adaptive schedule sets $\lambda_k$ high when the moving average of episode rewards is below a threshold and decays it exponentially once performance improves, so the stabilizing bias fades exactly as learning succeeds.

Load-bearing premise

The convergence proof assumes the regularized critic loss is strongly convex, but a neural-network critic with a temporal-difference error and a Sinkhorn penalty is not strongly convex in general, so the O(1/k) rate is not guaranteed for the implemented algorithm.

Editorial extensions

If this is right

  • Any actor-critic implementation can incorporate WAVE by evaluating one extra Sinkhorn cost per critic update; the actor update and policy-gradient structure remain unchanged.
  • If Theorem 1 holds, the regularized critic's squared parameter error is guaranteed to shrink as $C/k$, so the added penalty does not slow the asymptotic rate while it bounds the size of each update.
  • If Theorem 3 is correct, value-function estimates contract at factor $\gamma(1 - c\lambda)$ instead of $\gamma$, making WAVE strictly faster than unregularized Bellman iteration.
  • The reward-driven $\lambda$ schedule protects early training with high regularization and relaxes it automatically as performance improves, reducing the bias introduced by the penalty over time.
  • On the three continuous-control benchmarks reported, WAVE yields cumulative average reward above the TD3 baseline, including a stated gain of roughly 25% in the 2D robot-navigation task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The O(1/k) rate is the same rate that strongly convex stochastic gradient descent gives without any Wasserstein penalty, so the theoretical reason to prefer WAVE would be a better constant or lower gradient variance rather than a faster exponent; I would want a direct measurement of squared-error constants with and without the penalty to test that.
  • The regularizer links iterations $k$ and $k-1$ through the previous Q-distribution, so the recursion analyzed in Theorem 1 is not a standard first-order Markov gradient method; a rigorous extension would require a two-timescale or delayed-gradient analysis that the paper does not provide.
  • The adaptive $\lambda$ scheme depends on episodic rewards, so WAVE may need modification for sparse or delayed reward tasks, a limitation the paper acknowledges; testing it under reward-shaping or with a learned performance estimator is a natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes WAVE, a modification of actor-critic deep reinforcement learning in which the critic's loss is augmented by a Sinkhorn-approximated Wasserstein penalty between consecutive Q-value distributions, with an adaptively tuned regularization coefficient. The paper claims three contributions: an O(1/k) convergence-rate guarantee for the critic's mean squared error, a stability guarantee from the Wasserstein regularization, and empirical improvements over a TD3 baseline on Inverted Pendulum, Acrobot, and a custom 2D Robot Navigation task. The theoretical section presents three theorems, and the experiments report cumulative-average reward curves.

Significance. The motivating idea, penalizing large shifts in the Q-value distribution to stabilize actor-critic training, is reasonable and relevant, and the paper is clearly written. The authors are also candid in Section 5 about the limitations of their assumptions. However, the theoretical statements are the central contribution, and each of the three theorems has a load-bearing gap: Theorem 1 relies on strong convexity that the implemented neural-network critic does not satisfy, Theorem 2 assumes the very negative covariance it claims to establish, and Theorem 3 contains an invalid algebraic step in the contraction argument. The experimental section is too thin to substitute for the theory: it reports single curves without error bars or seeds, compares against only one baseline, and provides no code or reproducible artifacts. If the regularization idea were analyzed in a setting where it provably holds, or if the empirical study were substantially expanded, the work could be of interest; as it stands, the central claims are not supported.

major comments (5)
  1. [Section 3, Theorem 1] The central O(1/k) convergence claim is not established for the implemented algorithm. The proof assumes that the regularized loss Lreg(θQ)=LTD(θQ)+λk Wε(Q̂k, Q̂k−1) is strongly convex with parameter m>0. The implemented critic is a twin Q-network with three hidden layers of 256 units and LayerNorm; LTD is nonconvex in θQ, and the Sinkhorn penalty is convex in the Q-value vector, not in the network parameters, so there is no reason for the sum to be strongly convex. Section 5 concedes that the theoretical guarantees 'may not be universally applicable across all neural network architectures or RL tasks.' In addition, the theorem bounds E[||θk−θ*||²], whereas the abstract promises a rate for the critic's mean squared error; no Lipschitz conversion from parameter error to Q-value MSE is supplied, so even the stated conclusion does not match the abstract's claim.
  2. [Section 3, Theorem 1 proof] Even if strong convexity were granted, the proof is incomplete. The recursion φ_{k+1} ≤ (1 − 2am/k)φ_k + a²G²/k² yields the quoted O(1/k) rate only when 2am > 1; the paper never states this constraint on the learning-rate constant a and the strong-convexity constant m. For 2am ≤ 1, the standard analysis gives a slower rate, and no such case is treated. The proof also invokes 'the Lipschitz continuity of ∇θQ Lreg' in order to bound the cross term, but that assumption is not part of the theorem statement; the stated assumptions only include Lipschitz continuity of Q and boundedness of the gradient of Lreg.
  3. [Section 3, Theorem 2] The stability guarantee is circular. The proof computes Var(gTD + λk gW) = Var(gTD) + λk² Var(gW) + 2λk Cov(gTD, gW) and then assumes Cov(gTD, gW) ≤ 0 'because gW generally opposes changes suggested by gTD.' That negative correlation is precisely the effect the theorem claims to establish, and no independent argument or empirical evidence is provided. The additional bound ||∇θQ Lreg|| ≤ GTD + λk GW only controls the magnitude of the gradient, not the variance of the stochastic parameter updates, so it does not support the variance-reduction conclusion.
  4. [Section 3, Theorem 3] The contraction argument contains an invalid step. From |TλQ1(s,a) − TλQ2(s,a)| ≤ γ E_{s′,a′}[|Q1(s′,a′) − Q2(s′,a′)|] − λc1|Q1(s,a) − Q2(s,a)|, the proof immediately concludes |TλQ1(s,a) − TλQ2(s,a)| ≤ (γ − λc1)|Q1(s,a) − Q2(s,a)|. This is not a valid inequality: the first term is an expectation over next states and actions, which cannot be identified with the pointwise value |Q1(s,a) − Q2(s,a)|. The preceding derivative-difference bound is also merely asserted ('By properties of the Sinkhorn distance and Lipschitz continuity') and is not proved. Consequently, the claimed contraction factor γλ = γ(1 − cλ) is not derived, and the acceleration claim is unsupported.
  5. [Section 4] The empirical claim of superior performance is not supported by the reported evidence. The paper shows one cumulative-average-reward curve per environment, without standard deviations, number of seeds, or confidence intervals, and it compares only against a single baseline (TD3). The text gives a numeric improvement claim for the Robot Navigation environment, but the figure does not provide the aggregate data needed to verify it. Given that the theoretical guarantees do not apply to the implemented network, these experiments are the main evidence for WAVE, and as reported they are insufficient to support the abstract's statement that WAVE 'achieves superior performance compared to standard actor-critic methods.'
minor comments (5)
  1. [Section 2, Lemma 1] Lemma 1 is stated without proof or citation. The Sinkhorn distance is a biased approximation of the Wasserstein distance for ε>0, and differentiability holds only under conditions on the measures and cost; the statement should be made precise and referenced.
  2. [Section 2, Lemma 2] The statement that λk decreases as the agent's performance improves is not guaranteed by the update rule: λk is a function of a moving average R̄k, which can fluctuate, and when R̄k ≤ Rth the parameter is fixed at λmax rather than decreasing.
  3. [Section 2 and Section 3] The learning-rate notation is inconsistent: Section 2 uses ηk for the critic update, while Theorem 1 uses αk = a/k. The relationship between the analyzed schedule and the update actually used in Algorithm 1 is never specified.
  4. [Section 3, Theorem 1 proof] The proof says that a 'standard result' for the recursion yields φ_k ≤ C/k, but no reference or derivation is given. Please provide a proof or a citation, and also state the condition on 2am needed for the rate.
  5. [Section 4, Figure 2] The caption of Figure 2 shows the adaptation formula but the axis label uses λ(t) while the text uses λk; the notation should be unified, and the figure should indicate what is plotted (the value of the regularization parameter over episodes).

Circularity Check

2 steps flagged · score 6.0 of 10

Stability theorems assume their own conclusion; the advertised O(1/k) rate is a generic strongly convex SGD bound that never uses the Wasserstein term.

  1. other [Section 3, Theorem 2 proof]
    "Var(gTD + λkgW) = Var(gTD) + λ2kVar(gW) + 2λkCov(gTD, gW). Assuming that the covariance Cov(gTD, gW) ≤ 0 (because gW generally opposes changes suggested by gTD), the overall variance is reduced."

    The theorem's conclusion is that the Wasserstein term reduces the variance of critic updates and thereby stabilizes learning. The proof's only mechanism for that reduction is the unproved assumption Cov(gTD,gW)≤0; the displayed variance identity is a definitional expansion. If the covariance were positive, the same identity would show that variance increases. The claimed stability is therefore the assumed non-positive correlation restated, not a derived property of the Wasserstein regularizer.

  2. other [Section 3, Theorem 3 proof]
    "By properties of the Sinkhorn distance and Lipschitz continuity, we have |∂Wε( ˆQ1,k, ˆQ1,k−1) / ∂Q1(s, a) − ∂Wε( ˆQ2,k, ˆQ2,k−1) / ∂Q2(s, a)| ≥ c1|Q1(s, a) − Q2(s, a)|, for some constant c1 > 0."

    The improved contraction factor γλ = γ(1−cλ) is derived directly from the asserted lower bound with c1>0. That lower bound states that the Wasserstein-derivative term shrinks differences between Q-functions by at least a constant factor, which is already the enhanced contraction the theorem claims to prove. No lemma or external result establishes c1 from the Sinkhorn definition or the MDP, so the proof assumes the theorem's content and relabels it as a 'property of the Sinkhorn distance and Lipschitz continuity.'

full rationale

The paper's advertised theoretical guarantees separate into two parts. First, Theorem 1 is a standard strongly convex stochastic-gradient rate argument: it assumes strong convexity of Lreg(θQ), Lipschitzness, bounded gradients, and step sizes a/k, then derives E||ek||² = O(1/k). This is not itself circular, but it is generic and contains no Wasserstein term, λk, or Sinkhorn quantity; hence it does not establish that WAVE's specific regularization causes the rate. As written it also omits the condition 2am>1 required for the displayed recursion φ_{k+1} ≤ (1−2am/k)φ_k + a²G²/k² to yield O(1/k). Second, the stability results are circular in the narrower sense of assuming the conclusion. Theorem 2 proves variance reduction only by assuming Cov(gTD,gW)≤0, which is exactly the stabilizing opposition the theorem claims to demonstrate. Theorem 3 proves the stronger contraction γ(1−cλ) only by assuming a lower-bound inequality on Wasserstein-derivative differences that already encodes enhanced contraction; no argument derives c1 from the Sinkhorn distance, the critic architecture, or the MDP. These are not cases of fitted parameters being relabeled as predictions, and no load-bearing self-citation appears in the derivations; the WAVE-specific content is not imported from the authors' prior work. However, because the central stability guarantees reduce to unproved assumptions that match their conclusions, a moderate circularity score is warranted, while the empirical comparison against TD3 remains independently meaningful.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central theoretical claims rest on a strongly convex loss (unrealistic for neural critics), a negative covariance assumption that is the claimed conclusion, and an unproved Lipschitz lower bound on Sinkhorn derivatives. The method introduces several hand-set hyperparameters (λmax, λmin, ε) and several unreported ones (Rth, α, M). No new physical or mathematical entities are postulated; the regularized Bellman operator Tλ is a formal construction but is not connected to the actual gradient update.

free parameters (6)
  • λmax = 2.0
    Initial and maximum regularization weight, chosen by hand for all experiments (Section 4).
  • λmin = 0.3
    Minimum regularization weight, set by hand (Section 4).
  • ε (Sinkhorn) = 0.005
    Entropic regularization parameter in the Sinkhorn approximation, fixed by hand (Section 4).
  • Rth (performance threshold) = not reported
    Threshold in the adaptive λk update rule; no value is given in the paper (Section 2.1).
  • α (decay rate) = not reported
    Decay rate for λk when average reward exceeds Rth; no value is given (Section 2.1).
  • M (moving average window) = not reported
    Number of episodes used to form the moving average of cumulative reward; no value is given (Section 2.1).
assumptions (6)
  • ad hoc to paper The regularized critic loss Lreg(θQ) is strongly convex with parameter m > 0.
    Invoked in Theorem 1 proof (Section 3) to obtain the O(1/k) rate; not satisfied by typical neural network critics and acknowledged as a limitation in Section 5.
  • ad hoc to paper Gradients of the regularized loss are bounded: ||∇Lreg|| ≤ G.
    Standard SGD analysis assumption used in Theorem 1; not verified for the network and loss used in experiments.
  • domain assumption The critic Q(s,a;θQ) is Lipschitz continuous in θQ.
    Used in Theorem 1 and Theorem 2 to bound error growth; reasonable for bounded networks but never verified.
  • ad hoc to paper Cov(g_TD, g_W) ≤ 0, where g_TD and g_W are the TD and Wasserstein loss gradients.
    Assumed in Theorem 2 to conclude that regularization reduces variance; this is the conclusion the theorem is meant to prove (Section 3, Theorem 2).
  • ad hoc to paper The difference of Sinkhorn distance derivatives satisfies |∂Wε/∂Q1 - ∂Wε/∂Q2| ≥ c1 |Q1 - Q2| for some c1 > 0.
    Invoked in Theorem 3 to obtain a stronger contraction factor; no proof or empirical support is given, and the derivative is not well-defined.
  • domain assumption The transition kernel is Lipschitz continuous.
    Used in Theorem 3 to convert the contraction to a pointwise bound; unstated but implied in the proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning." pith.science (2026). https://pith.science/paper/LMJD7D2S

@misc{pith2026250110605,
  author       = {Pith},
  title        = {Pith review of: Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMJD7D2S}},
  note         = {Machine review of arXiv:2501.10605}
}
abstract

We present Wasserstein Adaptive Value Estimation for Actor-Critic (WAVE), an approach to enhance stability in deep reinforcement learning through adaptive Wasserstein regularization. Our method addresses the inherent instability of actor-critic algorithms by incorporating an adaptively weighted Wasserstein regularization term into the critic's loss function. We prove that WAVE achieves $\mathcal{O}\left(\frac{1}{k}\right)$ convergence rate for the critic's mean squared error and provide theoretical guarantees for stability through Wasserstein-based regularization. Using the Sinkhorn approximation for computational efficiency, our approach automatically adjusts the regularization based on the agent's performance. Theoretical analysis and experimental results demonstrate that WAVE achieves superior performance compared to standard actor-critic methods.

Figures

Figures reproduced from arXiv: 2501.10605 by the authors.

Figure 1
Figure 1. Comparison of cumulative average rewards between WAVE and baseline across three continuous control environments: Inverted Pendulum (left), Acrobot (middle), and 2D Robot Navigation (right) [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Evolution of the adaptive regularization parameter during training on the Inverted Pendulum environment. The Acrobot environment results further validate WAVE’s effectiveness. The proposed method achieves a final aver￾age reward of approximately -100, outperforming the base￾line’s reward. In the custom 2D Robot Navigation environ￾ment WAVE exhibits the en￾hanced final performance. The empirical results show an ap￾pr… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

  1. [1]

    Reinforcement Learning with Wasserstein Distance Regularisation, with Applications to Multipolicy Learning

    Abdullah, M.A., Pacchiano, A., Draief, M.: Reinforcement learning with wasserstein distance regularisation, with applications to multipolicy learning. arXiv preprint arXiv:1802.03976 (2018)

    work page Pith review arXiv 2018

  2. [2]

    arXiv preprint arXiv:1907.13196 (2019)

    Abdullah, M.A., Ren, H., Ammar, H.B., Milenkovic, V., Luo, R., Zhang, M., Wang, J.: Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196 (2019)

    arXiv 2019

  3. [3]

    arXiv preprint arXiv:2309.06239 (2023)

    Baheri, A.: Risk-aware reinforcement learning through optimal transport theory. arXiv preprint arXiv:2309.06239 (2023)

    arXiv 2023

  4. [4]

    arXiv preprint arXiv:2401.10949 (2024)

    Baheri, A., Kochenderfer, M.J.: The synergy between optimal transport theory and multi-agent reinforcement learning. arXiv preprint arXiv:2401.10949 (2024)

    arXiv 2024

  5. [5]

    Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithm

    Cayci, S., He, N., Srikant, R.: Finite-time analysis of entropy-regularized neural natural actor-critic algorithm. arXiv preprint arXiv:2206.00833 (2022)

    work page Pith review arXiv 2022

  6. [6]

    In: International conference on machine learning

    Fujimoto, S., Hoof, H., Meger, D.: Addressing function approximation error in actor- critic methods. In: International conference on machine learning. pp. 1587–1596. PMLR (2018)

    work page 2018

  7. [7]

    IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews)42(6), 1291– 1307 (2012)

    Grondman, I., Busoniu, L., Lopes, G.A., Babuska, R.: A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews)42(6), 1291– 1307 (2012)

    work page 2012

  8. [8]

    Mathematical Finance 33(3), 437–503 (2023)

    Hambly, B., Xu, R., Yang, H.: Recent advances in reinforcement learning in finance. Mathematical Finance 33(3), 437–503 (2023)

    work page 2023

Show all 18 references
  1. [9]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    He, S., Jiang, Y., Zhang, H., Shao, J., Ji, X.: Wasserstein unsupervised reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 6884–6892 (2022)

    2022

  2. [10]

    arXiv preprint arXiv:2006.00945 (2020)

    Hou, L., Pang, L., Hong, X., Lan, Y., Ma, Z., Yin, D.: Robust reinforcement learning with wasserstein constraint. arXiv preprint arXiv:2006.00945 (2020)

    2020 arXiv

  3. [11]

    The International Journal of Robotics Research32(11), 1238–1274 (2013) Wasserstein Adaptive Value Estimation 13

    Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: A survey. The International Journal of Robotics Research32(11), 1238–1274 (2013) Wasserstein Adaptive Value Estimation 13

    2013

  4. [12]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Likmeta, A., Sacco, M., Metelli, A.M., Restelli, M.: Wasserstein actor-critic: directed exploration via optimism for continuous-actions control. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 8782–8790 (2023)

    2023

  5. [13]

    Advances in Neural Information Processing Systems 32 (2019)

    Metelli, A.M., Likmeta, A., Restelli, M.: Propagating uncertainty in reinforcement learning via wasserstein barycenters. Advances in Neural Information Processing Systems 32 (2019)

    2019

  6. [14]

    Engineering Applications of Artificial Intelligence136, 108911 (2024)

    Razzaghi, P., Tabrizian, A., Guo, W., Chen, S., Taye, A., Thompson, E., Bregeon, A., Baheri, A., Wei, P.: A survey on reinforcement learning in aviation applications. Engineering Applications of Artificial Intelligence136, 108911 (2024)

    2024

  7. [15]

    arXiv preprint arXiv:1712.07185 (2017)

    Richemond, P.H., Maginnis, B.: On wasserstein reinforcement learning and the Fokker-Planck equation. arXiv preprint arXiv:1712.07185 (2017)

    2017 arXiv

  8. [16]

    arXiv preprint arXiv:2406.11774 (2024)

    Shahrooei, Z., Baheri, A.: Optimal transport-assisted risk-sensitive Q-learning. arXiv preprint arXiv:2406.11774 (2024)

    2024 arXiv

  9. [17]

    ACM Computing Surveys (CSUR)55(1), 1–36 (2021)

    Yu, C., Liu, J., Nemati, S., Yin, G.: Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR)55(1), 1–36 (2021)

    2021

  10. [18]

    Advances in Neural Information Processing Systems34, 15993–16006 (2021)

    Zhang, Y., Chen, S., Yang, Z., Jordan, M., Wang, Z.: Wasserstein flow meets replicator dynamics: A mean-field analysis of representation learning in actor-critic. Advances in Neural Information Processing Systems34, 15993–16006 (2021)

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.