pith. sign in

arxiv: 2606.04749 · v1 · pith:L7KCFHCAnew · submitted 2026-06-03 · 💻 cs.RO · cs.LG

COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection

Guopeng Li , Moritz A. Zanger , Matthijs T. J. Spaan , Julian F. P. Kooij This is my paper

Pith reviewed 2026-06-28 06:23 UTC · model grok-4.3

classification 💻 cs.RO cs.LG
keywords safe reinforcement learningrobot controlCholesky factorizationQ-learningjoint value estimationsafety constraintssample efficiency
0
0 comments X

The pith

COP-Q projects joint reward-safety Q-values via Cholesky factorization to reduce reward conservatism while preserving safety ordering in off-policy RL.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that separate handling of reward and safety critics in safe RL creates excessive conservatism because it ignores correlations between the two objectives. By building a joint confidence bound and factoring it with Cholesky to enforce a safety-first sequential order, COP-Q keeps the required safety margin while loosening the reward margin in a data-driven way. The adjusted vector-valued targets are then used for both critic updates and actor optimization. This change is presented as sufficient to recover sample efficiency without new computational cost or loss of safety guarantees. Experiments in locomotion and navigation domains are offered as evidence that the ordering is respected in practice.

Core claim

COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization.

What carries the argument

Cholesky-Ordered Projection applied to the joint reward-safety Q-value vector, which factors the covariance matrix to enforce a sequential safety-first ordering on the projected bounds.

If this is right

  • The same Cholesky projection step can be inserted into most existing deep Q-learning pipelines with only minor code changes.
  • The method applies equally to hard-constraint and soft-constraint formulations of safety.
  • Safety performance remains at least as strong as separate-critic baselines while sample efficiency is competitive or better.
  • Computational overhead stays negligible because the factorization is performed once per update on a low-dimensional objective vector.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same joint-space ordering idea could be tested on other pairs of objectives whose correlation structure is known in advance, such as energy and speed in locomotion.
  • If the Cholesky ordering proves stable, it may offer a lightweight alternative to full multi-objective Pareto-front methods when only a strict priority ordering is required.
  • The approach implicitly assumes that the critic ensembles produce well-calibrated joint uncertainty; any future work on better joint uncertainty estimation would directly strengthen COP-Q.

Load-bearing premise

The covariance captured by the Cholesky factor will correctly loosen the reward bound without weakening the safety bound or destabilizing the TD targets and actor updates.

What would settle it

A controlled run on the same Brax or Safety-Gymnasium tasks in which COP-Q either violates the safety threshold more often than the baselines or requires substantially more samples to reach the same return level would falsify the central claim.

Figures

Figures reproduced from arXiv: 2606.04749 by Guopeng Li, Julian F. P. Kooij, Matthijs T. J. Spaan, Moritz A. Zanger.

Figure 1
Figure 1. Figure 1: Obtaining a pair of Qπ c (safety cost) and Qπ r (reward) estimates without overestimation from multiple Q-value predictions by a set of critics. (a) Treating two objectives independently may give an over-conservative es￾timate and reduce sample efficiency, causing a safe policy with a low return; (b) Scalarization-based approaches may overestimate safety and lead to an unsafe policy; (c) The proposed Chole… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the two experimental environments. (a) Locomotion: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Benchmark on Safe Velocity CRL tasks. COP-Q is compared with [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Benchmark on Safe Navigation CRL tasks. COP-Q is compared [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison between the independent double-Q baseline and COP [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
read the original abstract

Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes Cholesky-Ordered Projection Q-learning (COP-Q) for off-policy safe RL. It learns joint reward and safety Q-values, constructs a generalized confidence bound in vector Q-space, and applies Cholesky factorization to encode safety-first priority sequentially. This is used for TD targets and actor optimization, with the goal of preserving safety conservatism while adaptively relaxing reward conservatism via inter-objective covariance. Experiments on Brax locomotion and Safety-Gymnasium navigation (hard and soft constraints) report strong safety performance with competitive or improved sample efficiency versus baselines.

Significance. If the projection mechanism holds, the work offers a lightweight, framework-compatible extension to existing safe RL methods that directly addresses over-conservatism from independent uncertainty handling. The emphasis on minimal overhead and compatibility with deep Q-learning frameworks is a practical strength.

major comments (2)
  1. [Method description of generalized confidence bound and Cholesky projection] The central claim that sequential Cholesky factorization in joint Q-space preserves safety-first ordering while reducing reward conservatism depends on the empirical covariance yielding a well-conditioned factorization. No formal analysis or bound is provided showing that noisy or mis-estimated covariances (common early in training or under function approximation) cannot invert the ordering or inflate variance in the projected Q-values used for both critic and actor updates.
  2. [Sections on TD target computation and actor optimization] The TD target stability and actor optimization claims rest on the projected Q-values not introducing bias from the factorization. The manuscript provides no derivation or sensitivity analysis demonstrating that the projection leaves the safety constraint satisfaction intact when inter-objective correlations are high, which is load-bearing for the safety guarantee.
minor comments (2)
  1. [Abstract] The abstract states 'minimal computational overhead' without any runtime or complexity comparison to the separate-critic baselines.
  2. [Method] Notation for the joint Q-vector and the exact form of the generalized confidence bound should be introduced with explicit equations rather than high-level description.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major comment below, providing clarifications on the empirical nature of our contributions while indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Method description of generalized confidence bound and Cholesky projection] The central claim that sequential Cholesky factorization in joint Q-space preserves safety-first ordering while reducing reward conservatism depends on the empirical covariance yielding a well-conditioned factorization. No formal analysis or bound is provided showing that noisy or mis-estimated covariances (common early in training or under function approximation) cannot invert the ordering or inflate variance in the projected Q-values used for both critic and actor updates.

    Authors: We agree that the manuscript lacks a formal analysis or theoretical bounds on the effects of noisy covariance estimates. The Cholesky-Ordered Projection is motivated by encoding sequential priority to maintain safety conservatism, with inter-objective covariance used to adaptively relax reward conservatism. Our experiments across Brax locomotion and Safety-Gymnasium tasks (including early training phases) show consistent safety performance without observed ordering inversions. However, we acknowledge this as a limitation of the current analysis. In the revised version, we will add a dedicated limitations subsection discussing covariance estimation sensitivity and include supplementary experiments varying covariance update rates to empirically assess stability. revision: yes

  2. Referee: [Sections on TD target computation and actor optimization] The TD target stability and actor optimization claims rest on the projected Q-values not introducing bias from the factorization. The manuscript provides no derivation or sensitivity analysis demonstrating that the projection leaves the safety constraint satisfaction intact when inter-objective correlations are high, which is load-bearing for the safety guarantee.

    Authors: The projection is constructed to enforce safety-first ordering via the Cholesky factorization, which sequentially prioritizes the safety objective before adjusting the reward component based on covariance. This design aims to preserve constraint satisfaction even under correlation, as validated in our hard-constraint Safety-Gymnasium experiments where COP-Q meets safety thresholds comparably to baselines. We do not claim or derive a formal guarantee of zero bias or invariance under arbitrary high correlations, as the approach is algorithmic and relies on empirical validation rather than theoretical bounds. We will revise the discussion section to explicitly state that safety performance is demonstrated empirically and note the absence of formal sensitivity analysis as an avenue for future work. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation builds on standard RL components without reduction to inputs.

full rationale

The paper presents COP-Q as a novel construction that augments standard off-policy Q-learning with joint covariance via Cholesky factorization to encode safety-first ordering. No load-bearing step reduces by definition or self-citation to the target result; the generalized confidence bound and sequential projection are defined from first principles on the vector Q-values, with TD targets and actor updates following directly. Experiments on Brax and Safety-Gymnasium provide external validation rather than internal fitting. The central claim therefore remains independent of its own fitted values or prior self-referential theorems.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract mentions no explicit free parameters, new entities, or ad-hoc axioms beyond standard RL assumptions.

axioms (1)
  • domain assumption Off-policy Q-learning with separate or joint critics can be extended to vector-valued estimates while preserving convergence properties.
    The method relies on existing deep Q-learning frameworks and TD updates.

pith-pipeline@v0.9.1-grok · 5741 in / 1164 out tokens · 24946 ms · 2026-06-28T06:23:46.383032+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 1 canonical work pages

  1. [1]

    Safe reinforcement learning for legged locomotion,

    T.-Y . Yang, T. Zhang, L. Luu, S. Ha, J. Tan, and W. Yu, “Safe reinforcement learning for legged locomotion,” inProc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2022, pp. 2454–2461

    2022

  2. [2]

    Dense reinforcement learning for safety validation of autonomous vehicles,

    S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, and H. X. Liu, “Dense reinforcement learning for safety validation of autonomous vehicles,”Nature, vol. 615, no. 7953, pp. 620–627, 2023

    2023

  3. [3]

    Safe learning in robotics: From learning-based control to safe reinforcement learning,

    L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,”Annu. Rev. Control Robot. Auton. Syst., vol. 5, no. 1, pp. 411–444, 2022

    2022

  4. [4]

    Issues in using function approximation for reinforcement learning,

    S. Thrun and A. Schwartz, “Issues in using function approximation for reinforcement learning,” inProc. 4th Connectionist Models Summer School, 1993, pp. 255–263

    1993

  5. [5]

    Deep reinforcement learning with double Q-learning,

    H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” inProc. AAAI Conf. Artif. Intell., vol. 30, 2016, pp. 2094–2100

    2016

  6. [6]

    A review of uncertainty for deep rein- forcement learning,

    O. Lockwood and M. Si, “A review of uncertainty for deep rein- forcement learning,” inProc. AAAI Conf. Artif. Intell. Interact. Digit. Entertainment, vol. 18, 2022, pp. 155–162

    2022

  7. [7]

    Constraints penalized Q-learning for safe offline reinforcement learning,

    H. Xu, X. Zhan, and X. Zhu, “Constraints penalized Q-learning for safe offline reinforcement learning,” inProc. AAAI Conf. Artif. Intell., vol. 36, 2022, pp. 8753–8760

    2022

  8. [8]

    Scalarized multi- objective reinforcement learning: Novel design techniques,

    K. van Moffaert, M. M. Drugan, and A. Nowe, “Scalarized multi- objective reinforcement learning: Novel design techniques,” inProc. IEEE Symp. Adapt. Dyn. Program. Reinf. Learn. (ADPRL), 2013, pp. 191–199

    2013

  9. [9]

    Safe reinforcement learning in constrained Markov decision processes,

    A. Wachi and Y . Sui, “Safe reinforcement learning in constrained Markov decision processes,” inProc. 37th Int. Conf. Mach. Learn. (ICML), vol. 119, 2020, pp. 9797–9806

    2020

  10. [10]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” inProc. 34th Int. Conf. Mach. Learn. (ICML), vol. 70, 2017, pp. 22–31

    2017

  11. [11]

    Responsive safety in reinforce- ment learning by PID Lagrangian methods,

    A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforce- ment learning by PID Lagrangian methods,” inProc. 37th Int. Conf. Mach. Learn. (ICML), vol. 119, 2020, pp. 9133–9143

    2020

  12. [12]

    Constrained variational policy optimization for safe reinforcement learning,

    Z. Liu, Z. Cen, V . Isenbaev, W. Liu, Z. S. Wu, B. Li, and D. Zhao, “Constrained variational policy optimization for safe reinforcement learning,” inProc. 39th Int. Conf. Mach. Learn. (ICML), vol. 162, 2022, pp. 13 644–13 668

    2022

  13. [13]

    Off-policy primal-dual safe reinforcement learning,

    Z. Wu, B. Tang, Q. Lin, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang, “Off-policy primal-dual safe reinforcement learning,” inInt. Conf. Learn. Represent. (ICLR), 2024

    2024

  14. [14]

    Optimistic exploration for risk-averse constrained reinforcement learning,

    J. McCarthy, R. Marinescu, E. Daly, and I. Dusparic, “Optimistic exploration for risk-averse constrained reinforcement learning,” in Proc. 28th Eur. Conf. Artif. Intell. (ECAI), 2025, pp. 1977–1984

    2025

  15. [15]

    Double Q-learning,

    H. van Hasselt, “Double Q-learning,” inAdv. Neural Inf. Process. Syst., vol. 23, 2010

    2010

  16. [16]

    Addressing function approximation error in actor-critic methods,

    S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” inProc. 35th Int. Conf. Mach. Learn. (ICML), vol. 80, 2018, pp. 1587–1596

    2018

  17. [17]

    Randomized ensembled double Q-learning: Learning fast without a model,

    X. Chen, C. Wang, Z. Zhou, and K. W. Ross, “Randomized ensembled double Q-learning: Learning fast without a model,” inInt. Conf. Learn. Represent. (ICLR), 2021

    2021

  18. [18]

    A practical guide to multi-objective reinforcement learning and planning,

    C. F. Hayes, R. Radulescu, E. Bargiacchi, J. Kallstrom, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowe, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers, “A practical guide to multi-objective reinforcement learning and planning,”Auton. Agents Multi-Agent Syst., vol. 3...

    2022

  19. [19]

    Multi-objective reinforcement learning: Convexity, stationarity and Pareto optimality,

    H. Lu, D. Herman, and Y . Yu, “Multi-objective reinforcement learning: Convexity, stationarity and Pareto optimality,” inInt. Conf. Learn. Represent. (ICLR), 2023

    2023

  20. [20]

    Off-policy safe reinforcement learning with cost-constrained optimistic exploration,

    G. Li, M. T. J. Spaan, and J. F. P. Kooij, “Off-policy safe reinforcement learning with cost-constrained optimistic exploration,” inInt. Conf. Learn. Represent. (ICLR), 2026

    2026

  21. [21]

    SafeDreamer: Safe reinforcement learning with world models,

    W. Huang, J. Ji, C. Xia, B. Zhang, and Y . Yang, “SafeDreamer: Safe reinforcement learning with world models,” inInt. Conf. Learn. Represent. (ICLR), 2024

    2024

  22. [22]

    Gaussian approximation for bias reduction in Q- learning,

    C. D’Eramo, A. Cini, A. Nuara, M. Pirotta, C. Alippi, J. Peters, and M. Restelli, “Gaussian approximation for bias reduction in Q- learning,”J. Mach. Learn. Res., vol. 22, no. 277, pp. 1–51, 2021

    2021

  23. [23]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inProc. 35th Int. Conf. Mach. Learn. (ICML), vol. 80, 2018, pp. 1861–1870

    2018

  24. [24]

    New perturbation analyses for the Cholesky factorization,

    X.-W. Chang, C. C. Paige, and G. W. Stewart, “New perturbation analyses for the Cholesky factorization,”IMA J. Numer. Anal., vol. 16, no. 4, pp. 457–484, 1996

    1996

  25. [25]

    Brax–a differentiable physics engine for large scale rigid body simulation,

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem, “Brax: A differentiable physics engine for large scale rigid body simulation,”arXiv preprint arXiv:2106.13281, 2021

    work page arXiv 2021

  26. [26]

    Safety Gymnasium: A unified safe reinforcement learning benchmark,

    J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y . Geng, Y . Zhong, J. Dai, and Y . Yang, “Safety Gymnasium: A unified safe reinforcement learning benchmark,” inAdv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 18 964–18 993

    2023

  27. [27]

    OmniSafe: An infrastructure for accelerating safe reinforcement learning research,

    J. Ji, J. Zhou, B. Zhang, J. Dai, X. Pan, R. Sun, W. Huang, Y . Geng, M. Liu, and Y . Yang, “OmniSafe: An infrastructure for accelerating safe reinforcement learning research,”J. Mach. Learn. Res., vol. 25, no. 285, pp. 1–6, 2024

    2024

  28. [28]

    D. G. Luenberger and Y . Ye,Linear and Nonlinear Programming, 4th ed. Springer, 2016

    2016

  29. [29]

    Reward constrained policy optimization,

    C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” inInt. Conf. Learn. Represent. (ICLR), 2019

    2019

  30. [30]

    Constrained update projection approach to safe policy optimization,

    L. Yang, J. Ji, J. Dai, L. Zhang, B. Zhou, P. Li, Y . Yang, and G. Pan, “Constrained update projection approach to safe policy optimization,” inAdv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 9111–9124

    2022

  31. [31]

    Saute RL: Almost surely safe rein- forcement learning using state augmentation,

    A. Sootla, A. I. Cowen-Rivers, T. Jafferjee, Z. Wang, D. H. Mguni, J. Wang, and H. Bou-Ammar, “Saute RL: Almost surely safe rein- forcement learning using state augmentation,” inProc. 39th Int. Conf. Mach. Learn. (ICML), vol. 162, 2022, pp. 20 423–20 443

    2022