Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that overestimation bias in off-policy reinforcement learning can be controlled by jointly reweighting TD errors and predictive uncertainty from a distributional critic, and by sampling pessimistic values in the actor upda

desk verdict Broad empirical wins for a plausible plug-in, but the uncertainty mechanism is underspecified and the theory is mostly narrative; needs a serious revision before I'd trust the mechanism. read the letter →

arxiv 2607.26509 v1 pith:TYLN5EKD submitted 2026-07-29 cs.LG cs.AI

Gong Gao , Xiao Lai , Ziqi Xie , Guojie Chen , Xianhui Liu , Weidong Zhao This is my paper
classification cs.LGcs.AI
keywords off-policyreinforcementlearningoverestimationbiasdistributionalcriticpredictiveuncertaintycollaborativeweightingstochasticpessimisticvalueestimationcontinuouscontrolTD-error
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep off-policy reinforcement learning suffers from a self-reinforcing loop: temporal-difference targets carry noise, and greedy policy improvement picks the actions that were most overestimated, so errors grow. The paper proposes Collaborative Weighting Actor-Critic (CWAC), which attacks both ends of that loop. A distributional critic provides a mean and a standard deviation for each return estimate; a collaborative weighting scheme down-weights transitions with high uncertainty and also prevents the uncertainty estimate from collapsing prematurely on high-residual samples. The actor then maximizes a stochastically pessimistic value estimate rather than the raw Q-value, reducing the amplification caused by greedy updates. A sympathetic reader would care because, if this works, one can stabilize value learning and improve sample efficiency in continuous-control tasks by adding a small, portable module to existing off-policy algorithms rather than redesigning them.

What carries the argument

The central object is the collaborative weighting mechanism built on a distributional critic. The distributional critic outputs both mean Q and standard deviation sigma of the return distribution for a state-action pair. The critic objective weights each sample's Huber TD error by omega = (E[sigma]/(sigma+c))^beta_omega and adds a regularizer xi * sigma, with xi = (E[|delta|]/(|delta|+c))^beta_xi. The reciprocal design is what carries the argument: omega shrinks when uncertainty is high, while xi shrinks when the TD error is large, so the uncertainty estimate is not minimized to zero on hard samples. The second mechanism is stochastic pessimistic value sampling Z = Q - |epsilon| sigma, used

What would settle it

On a held-out set of transitions, compute the rank correlation between sigma_theta and the squared TD error, or between sigma_theta and the squared error against a long-rollout return estimate; if the correlation is near zero, the collaborative weighting is not targeting noisy samples and cannot be the cause of improved performance. A stronger variant: replace sigma_theta with random noise of matched scale in both the loss and the actor, and check whether the performance gap over the base algorithm disappears.

Watch

Extended reading notes

Core claim

The central claim: overestimation bias in off-policy actor-critic methods is best attacked by using the return distribution itself as both a learning weight and a pessimistic target. CWAC's critic loss combines a Huber TD error with the predicted standard deviation, with weight terms that down-weight high-uncertainty samples and prevent premature collapse of the uncertainty estimate. The actor maximizes a stochastic pessimistic value, making pessimism scale with uncertainty. Across 6 Gym, 4 PyBullet, and 12 DeepMind Control tasks, the paper reports the highest average return among five baselines, improved value-error stability under noisy rewards, and successful transfer to two deterministic

Load-bearing premise

The entire weighting scheme rests on the assumption that the standard deviation sigma_theta returned by the distributional critic actually tracks predictive uncertainty; the paper trains sigma through a weighted regression term and never validates the calibration, so if sigma merely reflects regression noise, high-uncertainty transitions will not be the ones down-weighted and the mechanism loses its justification.

Editorial extensions

If this is right

  • Overestimation can be suppressed without permanently conservative policies, because the pessimism is stochastic and scales with uncertainty, leaving room for exploration.
  • Purely TD-error-based prioritization is risky; the paper's paired weighting indicates such schemes should jointly account for uncertainty to avoid amplifying noisy transitions.
  • The recipe is portable: adding it to standard deterministic off-policy actor-critic methods improved convergence and final returns and did not degrade the remaining test tasks.
  • Value estimates stay closer to ground truth during training, as evidenced by smoother TD-error trajectories, which should translate to safer policy iteration in real systems.
  • A fixed set of three hyperparameters worked across diverse benchmarks, suggesting the mechanism does not require per-task tuning to be useful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If sigma is genuinely calibrated, the same collaborative weighting could be lifted into offline reinforcement learning, where noisy value targets are the central obstacle and the cost of overconfidence is higher.
  • The error-conditioned uncertainty optimization is a general trick: any uncertainty estimator trained by regression can collapse to overconfidence on hard samples, and a reciprocal weight that down-weights large residuals may prevent that collapse in other settings.
  • Extending the stochastic pessimistic sampling to multi-step returns or model-based value expansion could cut bias further, but the paper does not test these variants.
  • A direct calibration check on held-out transitions would predict where CWAC's advantage shows up; environments where sigma and true error are uncorrelated should show no gain from the weighting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Collaborative Weighting Actor-Critic (CWAC), an off-policy RL method intended to mitigate overestimation bias. CWAC augments a base actor-critic algorithm (SAC, TD3, or DDPG) with: (i) a critic that outputs a mean Qθ(s,a) and a standard deviation σθ(s,a); (ii) a stochastic pessimistic value estimator Zθ(s,a)=Qθ(s,a)−|ε|σθ(s,a) with ε∼N(0,µI), used both in the bootstrapped target (Eq. 6) and the actor objective (Eq. 8); and (iii) a collaborative weighting scheme (Eq. 7) that down-weights critic updates in high-uncertainty regions and adjusts the σ penalty based on TD-error magnitude. The authors report consistent performance improvements over VIAC, ALH, LAP, SAC, and TD3 across 22 Gym, PyBullet, and DMC tasks, with modest computational overhead, and provide sensitivity and value-error visualizations.

Significance. If the mechanism were fully established, CWAC would be a practical and general add-on to off-policy actor-critic algorithms, with the attractive property of suppressing overestimation while retaining exploration. The empirical study is broad (22 environments, 10 seeds, multiple base algorithms), and the paper includes sensitivity analyses, runtime comparisons, and TD-error diagnostics. However, the theoretical grounding is not yet rigorous: the role of σθ as a calibrated predictive uncertainty is asserted rather than derived, the loss in Eq. (5) contains no distributional or calibration objective for σθ, and the reported gradient in Intuition 2 is incorrect. These gaps directly affect the central claim that collaborative weighting is driven by uncertainty, so the contribution is currently more empirical than mechanistic.

major comments (4)
  1. [§4.2, Eq. (5)] The critic loss is L = E[ω·Huber(Qθ−y) + ξ·σ]. This is the only stated training signal for σθ. Since ∂L/∂σ = ξ > 0 everywhere (ignoring the dependence of ω on σ), gradient descent simply minimizes σ; there is no term that makes σθ approximate the conditional variance of the return, the Bellman uncertainty, or any distributional target. The paper calls the critic “distributional” but provides no quantile loss, return-distribution target, or calibration check. Because ω in Eq. (7) and Z in Eq. (4) both rely on σθ, the central mechanism loses its claimed justification: without a calibrated σθ, the weighting and pessimistic sampling are not demonstrably uncertainty-aware. The authors should specify an explicit distributional/uncertainty objective for σθ or provide empirical evidence that σθ tracks predictive uncertainty.
  2. [§4.2, Intuition 2] The stated gradient ∇L = E[2ω·δ·∇Q + ξ·∇σ] is incorrect as written. For the Huber loss, the Q-gradient is not 2ωδ in the linear-loss region. More importantly, ω depends on σ through ω=(Eσ/(σ+c))^βω, so the σ-gradient contains an additional term −βω·ω·Huber(Q−y)/(σ+c), which is omitted. The sign analysis for the “reciprocal interaction” is therefore incomplete; a term that decreases the weight on the Bellman error as σ grows could counteract or reinforce the direct ξ penalty. The authors should derive the full gradients and analyze the resulting equilibrium, or explicitly state under which approximations the simplified gradient holds.
  3. [§5.1, Table 5, §5.7] The default hyperparameter values for the two weighting exponents are mutually inconsistent. §5.1 states βω=2 and βξ=1; Table 5 lists βω=1 and βξ=2; §5.7 states “we set βω=1 and βξ=2 as the default setting.” Since these exponents directly determine the strength of the collaborative weighting mechanism, the experimental results are not reproducible without resolving which configuration was actually used. The sensitivity analysis also becomes ambiguous if the “default” differs between sections. Please correct the inconsistency and report the exact configuration used in each experiment.
  4. [§4.2, Theorem 1] Theorem 1 is presented as a formal result but is only a proof sketch, and it does not establish the paper’s actual claim. The sketch assumes η with E[η]>0 and argues that weights proportional to |δ|^p bias the update, but it neither proves that the bias “increases with the noise magnitude” nor connects this to the proposed collaborative weighting. As written, the theorem only motivates why naive prioritization can be problematic; it does not support the sufficiency of Eq. (5)–(8). This is acceptable as intuition, but it should be labeled as such or replaced with a result that actually links the proposed weighting to overestimation reduction.
minor comments (5)
  1. [§4.1, Intuition 1] The numeric example does not support the stated conclusion. With G(s,a1)=1, G(s,a2)=2, e=(0.01,0.1), b=(0.01,0.5), one obtains Z(s,a1)=1.0 and Z(s,a2)=1.6, so Z(a2)>Z(a1). The text claims “Z(s,a1)>Z(s,a2).” Either the numbers or the intended message need adjustment.
  2. [Abstract vs §7.4] The abstract states that code is publicly available at https://anonymous.4open.science/r/CWAC-348E, while §7.4 says “The code used in this study is available upon request.” Please reconcile these statements.
  3. [Table 5] In Table 5, the “Value function” row for CWAC reads “Huber expectile,” which is ambiguous. The text only mentions Huber loss in Eq. (5); clarify whether an expectile term is also used. The table also does not describe the architecture or update rule for the σθ head.
  4. [§5.6 / Table 5] The text claims a “unified hyperparameter configuration,” but Table 5 shows different start timesteps (1e4 for CWAC vs 25e3 for most baselines) and different policy update frequencies. While these differences may be inherited from the base algorithms, the claim of identical conditions should be qualified.
  5. [General] There are typographical and consistency issues: “Collaborative Weighting Actor-Criti” in §1, inconsistent use of “CW AC” and “CWAC,” and Figure 6 legend entries such as “=0” without a variable name. Please proofread and standardize notation.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the derivation is self-contained, with one minor non-load-bearing self-citation and a non-circular calibration gap in the uncertainty estimate.

full rationale

The paper's central performance claims are empirical: CWAC is evaluated on Gym/PyBullet/DMC against VIAC, ALH, LAP, SAC, and TD3, and the reported gains are not recovered by fitting any constant to those benchmarks. Walking the derivation chain, Eqs. (4)-(8) define an objective rather than derive an external quantity: the pessimistic target Z is defined from Q and sigma, the weights omega and xi are defined as functions of sigma and delta, and the critic/actor losses use these quantities directly. No step takes a fitted parameter and renames it a prediction, and no 'uniqueness theorem' or prior result by the same authors is used to force the design. The only self-citation, [25], appears in a related-work sentence alongside [24] and is not load-bearing. The substantive weakness--that Eq. (5) gives sigma only a xi*sigma penalty and no distributional/calibration objective, so 'predictive uncertainty' is not guaranteed to be calibrated--is an internal-validity/correctness risk rather than circularity, since the empirical comparison would still be meaningful even if the mechanistic label were wrong. Section 6 itself notes the conservatism limitation. Score 2 reflects only the minor non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

CWAC's contribution is the algorithm and its benchmark results. Every quantitative constant in the objective is hand-set, and the theoretical motivation rests on unproved assumptions about noise sign and uncertainty ordering. The uncertainty estimate itself is trained through the proposed loss, so it is not independently calibrated.

free parameters (4)
  • pessimism coefficient mu = 0.8
    Controls the magnitude of Gaussian noise subtracted in Eq. (4); hand-set and swept only on Walker2d in Section 5.7.
  • uncertainty exponent beta_omega = 1 (Table 5) or 2 (Section 5.1)
    Sharpness of uncertainty weighting in Eq. (7); the paper contradicts itself on the default value.
  • TD-error exponent beta_xi = 2 (Table 5) or 1 (Section 5.1)
    Sharpness of TD-error weighting in Eq. (7); the paper contradicts itself on the default value.
  • stability constant c = unspecified
    Used in the denominators of Eq. (7); its numerical value is never reported.
assumptions (4)
  • standard math Standard discounted MDP and soft Bellman framework with Gaussian policy (Eq. 1-2)
    Adopted from SAC and standard RL setting; not questioned by the paper.
  • domain assumption Uncertainty in unvisited state-action pairs is at least as large as in visited pairs: sigma_train <= sigma_test
    Used in Intuition 1 to argue excessive pessimism can reverse action preference; never verified or derived.
  • domain assumption TD-target noise eta has positive mean E[eta] > 0 in Theorem 1
    The proof sketch depends on noise being positively correlated with the prioritization weight; zero-mean noise would weaken the claimed bias.
  • ad hoc to paper The critic's standard-deviation head sigma_theta is a faithful return-uncertainty estimate
    No distributional loss or calibration target is defined; this is the load-bearing premise of Eq. (5)-(7).
invented entities (1)
  • Stochastic pessimistic value estimator Z_theta(s,a)=Q_theta(s,a)-|epsilon|*sigma
    purpose: Replaces Q in the actor and target updates to reduce maximization bias through random pessimism.
    This is a new algorithmic construct introduced by the paper, with no external falsifiable handle; its benefit is only shown in the same-benchmark reward curves and is not theoretically derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning." pith.science (2026). https://pith.science/paper/TYLN5EKD

@misc{pith2026260726509,
  author       = {Pith},
  title        = {Pith review of: Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYLN5EKD}},
  note         = {Machine review of arXiv:2607.26509}
}
read the original abstract

Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates amplify early-stage estimation errors. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor-critic methods. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying bias.In this paper, we propose Collaborative Weighting Actor-Critic (CWAC), a unified framework that explicitly accounts for predictive uncertainty in value estimation. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement. CWAC can be seamlessly integrated into existing off-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks. Our code is publicly available at https://anonymous.4open.science/r/CWAC-348E.

Figures

Figures reproduced from arXiv: 2607.26509 by the authors.

Figure 1
Figure 1. Illustration of two fundamental challenges in o [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Learning curves on 6 Gym tasks. The shaded region represents half a standard deviation of the [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Learning curves on four PyBullet tasks. The shaded areas represent half of the standard deviation [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Learning curves on 12 DMC tasks. The shaded region represents half a standard deviation of the [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Learning curves of CWAC, TD3, and DDPG on six Gym continuous control tasks. Shaded regions [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Hyperparameter sensitivity analysis. The shaded region represents half a standard deviation of the [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Value estimation error on 6 Gym tasks. The shaded region represents half a standard deviation of [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Comparison of evaluation performance and estimation error in noisy-reward and noise-free envi [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 10 linked inside Pith

  1. [1]

    H. Xiao, C. Chen, G. Zhang, C. P. Chen, Reinforcement learning-driven dynamic obstacle avoidance for mobile robot trajectory tracking, Knowledge-Based Sys- tems 297 (2024) 111974

    2024

  2. [2]

    D. Kim, S. Park, H. Jang, J. Shin, J. Kim, Y . Seo, Robot-r1: Reinforcement learning for enhanced embodied reasoning in robotics, Advances in Neural Infor- mation Processing Systems 38 (2026) 161472–161507. 26

    2026

  3. [3]

    H. Shi, Y . Zhou, K. Wu, S. Chen, B. Ran, Q. Nie, Physics-informed deep rein- forcement learning-based integrated two-dimensional car-following control strat- egy for connected automated vehicles, Knowledge-Based Systems 269 (2023) 110485

    2023

  4. [4]

    Z. Yang, X. Jia, Q. Li, X. Yang, M. Yao, J. Yan, Raw2drive: Reinforcement learn- ing with aligned world models for end-to-end autonomous driving (in carla v2), Advances in Neural Information Processing Systems 38 (2026) 134122–134147

    2026

  5. [5]

    Y . Liu, J. Fan, C. Zhang, W. Shen, A knowledge-driven deep reinforcement learn- ing approach for dynamic scheduling of re-entrant hybrid flow shop with in-line product quality inspection, Knowledge-Based Systems (2025) 114418

    2025

  6. [6]

    C. Ngwu, Y . Liu, R. Wu, Reinforcement learning in dynamic job shop schedul- ing: a comprehensive review of ai-driven approaches in modern manufacturing, Journal of Intelligent Manufacturing 37 (3) (2026) 1093–1108

    2026

  7. [7]

    L. Hua, H. Liu, Y . Pan, Solving industrial chain job scheduling problems through a deep reinforcement learning method with decay strategy, Information Sciences 702 (2025) 121906

    2025

  8. [8]

    Haarnoja, A

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, et al., Soft actor-critic algorithms and applications, arXiv preprint arXiv:1812.05905 (2018)

    arXiv 2018

Show all 40 references
  1. [9]

    Fujimoto, H

    S. Fujimoto, H. Hoof, D. Meger, Addressing function approximation error in actor-critic methods, in: International conference on machine learning, PMLR, 2018, pp. 1587–1596

    2018

  2. [10]

    Tesauro, et al., Temporal difference learning and td-gammon, Communications of the ACM 38 (3) (1995) 58–68

    G. Tesauro, et al., Temporal difference learning and td-gammon, Communications of the ACM 38 (3) (1995) 58–68

    1995

  3. [11]

    Cetin, O

    E. Cetin, O. Celiktutan, Learning pessimism for reinforcement learning, in: Pro- ceedings of the AAAI conference on artificial intelligence, V ol. 37, 2023, pp. 6971–6979. 27

    2023

  4. [12]

    Nauman, M

    M. Nauman, M. Bortkiewicz, P. Miło ´s, T. Trzcinski, M. Ostaszewski, M. Cygan, Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning, in: International Conference on Machine Learning, PMLR, 2024, pp. 37342–37364

    2024

  5. [13]

    J. Wang, L. Li, W. Wei, Y . Zhang, X. Yang, Dynamic uncertainty estimation for offline reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 39, 2025, pp. 21126–21134

    2025

  6. [14]

    T. G. Karimpanal, H. Le, M. Abdolshah, S. Rana, S. Gupta, T. Tran, S. Venkatesh, Balanced q-learning: Combining the influence of optimistic and pessimistic tar- gets, Artificial Intelligence 325 (2023) 104021

    2023

  7. [15]

    T. Tan, H. Xie, D. Lian, Adaptive order q-learning., in: International Joint Con- ference on Artificial Intelligence, 2024, pp. 4946–4954

    2024

  8. [16]

    Schaul, J

    T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized experience replay, arXiv preprint arXiv:1511.05952 (2015)

    2015 arXiv

  9. [17]

    Fujimoto, D

    S. Fujimoto, D. Meger, D. Precup, An equivalence between loss functions and non-uniform sampling in experience replay, Advances in neural information pro- cessing systems 33 (2020) 14219–14230

    2020

  10. [18]

    H. Zhu, P. Rashidinejad, J. Jiao, Importance weighted actor-critic for optimal conservative offline reinforcement learning, Advances in Neural Information Pro- cessing Systems 36 (2023) 49579–49602

    2023

  11. [19]

    Zhang, Z

    Z. Zhang, Z. Pan, M. J. Kochenderfer, Weighted double q-learning., in: Proceed- ings of the 26th International Joint Conference on Artificial Intelligence, 2017, pp. 3455–3461

    2017

  12. [20]

    Y . Wu, S. Zhai, N. Srivastava, J. Susskind, J. Zhang, R. Salakhutdinov, H. Goh, Uncertainty weighted actor-critic for offline reinforcement learning, arXiv preprint arXiv:2105.08140 (2021). 28

    2021 arXiv

  13. [21]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, nature 518 (7540) (2015) 529–533

    2015

  14. [22]

    Van Hasselt, A

    H. Van Hasselt, A. Guez, D. Silver, Deep reinforcement learning with double q-learning, in: Proceedings of the AAAI conference on artificial intelligence, V ol. 30, 2016

    2016

  15. [23]

    Q. Lan, Y . Pan, A. Fyshe, M. White, Maxmin q-learning: Controlling the estima- tion bias of q-learning, arXiv preprint arXiv:2002.06487 (2020)

    2002 arXiv

  16. [24]

    N. M. Quang, H. W. Lauw, Augmenting decision with hypothesis in reinforce- ment learning, in: Forty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    G. Gao, W. Zhao, X. Liu, N. Jia, Improving policy exploitation in online re- inforcement learning with instant retrospect action, Neural Networks (2026) 108667

    2026

  18. [26]

    Y . Oren, M. A. Zanger, P. R. V . der Vaart, M. M. Çelikok, W. Boehmer, M. T. J. Spaan, Value improved actor critic algorithms, in: The Thirty-ninth Annual Con- ference on Neural Information Processing Systems, 2025

    2025

  19. [27]

    C. Bai, L. Wang, Z. Yang, Z. Deng, A. Garg, P. Liu, Z. Wang, Pessimistic boot- strapping for uncertainty-driven offline reinforcement learning, arXiv preprint arXiv:2202.11566 (2022)

    2022 arXiv

  20. [28]

    X. Gong, S. Lü, J. Yu, S. Zhu, Z. Li, Adaptive estimation q-learning with un- certainty and familiarity, in: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023, pp. 3750–3758

    2023

  21. [29]

    K. Guo, S. Yunfeng, Y . Geng, Model-based offline reinforcement learning with pessimism-modulated dynamics belief, Advances in Neural Information Process- ing Systems 35 (2022) 449–461. 29

    2022

  22. [30]

    Moskovitz, J

    T. Moskovitz, J. Parker-Holder, A. Pacchiano, M. Arbel, M. Jordan, Tactical op- timism and pessimism for deep reinforcement learning, Advances in Neural In- formation Processing Systems 34 (2021) 12849–12863

    2021

  23. [31]

    X. B. Peng, A. Kumar, G. Zhang, S. Levine, Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learning, arXiv preprint arXiv:1910.00177 (2019)

    1910 arXiv

  24. [32]

    A. Nair, A. Gupta, M. Dalal, S. Levine, Awac: Accelerating online reinforcement learning with offline datasets, arXiv preprint arXiv:2006.09359 (2020)

    2006 arXiv

  25. [33]

    X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y . Gao, W. Li, L. Bin, C. Finn, C. Zhang, Lapo: Latent-variable advantage-weighted policy optimization for of- fline reinforcement learning, Advances in Neural Information Processing Systems 35 (2022) 36902–36913

    2022

  26. [34]

    Zhang, W

    S. Zhang, W. Zhang, Q. Gu, Energy-weighted flow matching for offline reinforce- ment learning, in: The Thirteenth International Conference on Learning Repre- sentations, 2025

    2025

  27. [35]

    Alles, N

    M. Alles, N. Chen, P. van der Smagt, B. Cseke, Flowq: Energy-guided flow poli- cies for offline reinforcement learning, arXiv preprint arXiv:2505.14139 (2025)

    2025 arXiv

  28. [36]

    Hassani, S

    H. Hassani, S. Nikan, A. Shami, Improved exploration–exploitation trade-off through adaptive prioritized experience replay, Neurocomputing 614 (2025) 128836

    2025

  29. [37]

    P. J. Huber, Robust estimation of a location parameter, in: Breakthroughs in statis- tics: Methodology and distribution, Springer, 1992, pp. 492–518

    1992

  30. [38]

    Brockman, V

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, W. Zaremba, Openai gym, arXiv preprint arXiv:1606.01540 (2016)

    2016 arXiv

  31. [39]

    Coumans, Y

    E. Coumans, Y . Bai, Pybullet, a python module for physics simulation for games, robotics and machine learning (2016). 30

    2016

  32. [40]

    Tassa, Y

    Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Ab- dolmaleki, J. Merel, A. Lefrancq, et al., Deepmind control suite, arXiv preprint arXiv:1801.00690 (2018). 31

    2018 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.