Pith. sign in

REVIEW 4 major objections 4 minor 75 references

Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Learning any deterministic policy's constraint-value function yields a discrete control barrier function, so one learned function can certify safety for a multi-agent system with unknown discrete-time dynamics.

desk verdict Solid discrete-time extension of policy-CBF to multi-agent RL; the central safety guarantee is empirically plausible but not theorem-backed at deployment. read the letter →

arxiv 2502.03640 v3 pith:FISYKRAP submitted 2025-02-05 cs.RO cs.LGcs.MAmath.OC

Songyuan Zhang , Oswin So , Mitchell Black , Chuchu Fan This is my paper · ORCID
classification cs.ROcs.LGcs.MAmath.OC
keywords discretecontrolbarrierfunctionmulti-agentreinforcementlearningsafeoptimalgraph-basedsafetycertificateunknowndiscrete-timedynamicsconstraint-valueproximalpolicyoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles multi-agent safe control when the dynamics are unknown, each agent sees only a limited neighborhood, and no high-performance nominal controller is available. Its central theoretical move is to prove that the worst-ever value of a safety constraint along a rollout, $V^{h,\mu}(x) = \max_{k\geq 0} h(x_k)$, is itself a discrete control barrier function for the deterministic policy that generated the rollout. That makes the safety certificate learnable by ordinary value-function regression alongside reinforcement learning, and the graph-structured version extends the certificate to changing neighborhoods. If the claim is right, safe multi-agent policies can be trained end-to-end without dynamics models, safety filters, or nominal controllers; the authors report lower costs with near-100% safety than penalty and Lagrangian baselines across seven environments using one fixed set of hyperparameters.

What carries the argument

The central object is the learned discrete graph CBF, $\tilde{B}(o_i) = V^{h,\mu}(o_i)$, the constraint-value function restricted to agent $i$'s local observation, defined as the maximum of $h$ along the deterministic rollout of policy $\mu$. It is a safety certificate because it obeys the one-step decrease condition $\tilde{B}(o_i^+) - \tilde{B}(o_i) + \alpha(\tilde{B}(o_i)) \leq 0$ whenever all agents start safe. It is parameterized as a graph attention network of the form $\xi_1(\sum_{j\in N_i} w(o_{ij})\xi_2(o_{ij}), \xi_3(o_i^y))$, where attention weights $w$ vanish for agents far enough away, which handles agents entering or leaving the sensing radius. Around it, the algorithm uses a CRPO-style primal update: if the constraint is satisfied, take the PPO cost step; otherwise take a score-function-gradient step minimizing the violation, with an indicator-based gradient projection (Informal Theorem 3, formalized as Theorem A2) intended to keep cost and safety gradient directions decoupled.

What would settle it

Sample a dense set of states from the safe set $\{x : \tilde{B}(o_i(x)) \leq 0\}$, apply the trained deterministic policy, and check the one-step inequality $\tilde{B}(o_i^+) - \tilde{B}(o_i) + \alpha(\tilde{B}(o_i)) \leq 0$; a single violation is a direct counterexample to the claim that the learned function is a DGCBF. Alternatively, compute $\nabla_\theta \log \pi_\theta(u|x)$ at two different states and measure their inner product, since nonzero inner products falsify the orthogonality condition behind Theorem A2.

Watch

Extended reading notes

Core claim

The load-bearing discovery is Theorem 2: for any fixed deterministic policy $\mu$ and any constraint function $h$, the function $V^{h,\mu}(x) = \max_{k\geq 0} h(x_k)$ along the rollout satisfies the discrete CBF inequality $V^{h,\mu}(f(x,\mu(x))) - V^{h,\mu}(x) + \alpha(V^{h,\mu}(x)) \leq 0$ whenever $V^{h,\mu}(x) \leq 0$, for any extended class-$\kappa$ function $\alpha$ with $\alpha(-r) > -r$. The proof is a one-line dynamic-programming identity, $V^{h,\mu}(x) = \max\{h(x), V^{h,\mu}(f(x,\mu(x)))\}$: whichever term reaches the max, the one-step difference is non-positive after adding $\alpha(V^{h,\mu}(x))$. Consequently the safety certificate does not have to be designed or separately learned; it is the value function of the very policy being trained, evaluated on the safety constraint. Corollary 1 lifts this to a discrete graph CBF on local observations, so the same function certifies safety when neighborhoods change, and Theorem A5 shows the same DGCBF extends to any number of agents under decoupled dynamics. The paper calls the resulting algorithm DGPPO.

Load-bearing premise

The load-bearing premise is that the policy's gradient at one state is orthogonal to its gradient at every other state, so that the safety and cost update directions separate cleanly; the paper says this cannot be guaranteed for the learned network policies, and if it fails, the claimed decoupling of the gradient projection is not established.

Editorial extensions

If this is right

  • Safety certificates can be learned by regression instead of designed: any learned policy determines a valid DCBF for its own deterministic version, so the framework removes the nominal-policy and known-dynamics assumptions that earlier CBF safety filters require.
  • The same DGCBF generalizes across agent counts: under per-agent decoupled dynamics and bounded per-step travel, a DGCBF certified for a moderate number of agents also certifies any larger number (Theorem A5), and the authors show deployment up to 512 agents with little performance loss.
  • Because the policy optimizes cumulative cost directly rather than being filtered myopically, the learned controller can avoid deadlocks that safety-filtered nominal policies suffer, as demonstrated in the paper's obstacle-avoidance comparison.
  • On seven benchmark tasks spanning different simulation engines, DGPPO with one hyperparameter set reaches lower cost than penalty and Lagrangian baselines while matching their best safety rates, and its training curves are smoother.
  • The expectation constraint $\mathbb{E}[\max\{0, C^{(m)}(x,u)\}] \leq 0$ is sufficient for almost-sure per-transition satisfaction of the DCBF condition (Theorem A1), which is why the CRPO-style update can use expected violations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the indicator projection with an explicit orthogonal projection of the Q-value gradient onto the nullspace of constraint gradients estimated from a batch; that would restore the cost-safety decoupling even when the paper's orthogonality assumption fails for shared-parameter network policies.
  • The construction suggests a general principle for constrained RL: any constraint evaluable along a rollout defines a certificate for the policy that produced the rollout, so safety can be certified after the fact at no extra modeling cost, a property that could transfer to imitation learning or model-predictive controllers.
  • Because the learned function only approximately satisfies the DGCBF inequality, a practical safety check is to search sampled safe states for violations of the one-step condition and certify with a margin; doing this would turn the empirical near-100% safety claim into a verifiable bound.
  • The framework assumes deterministic dynamics; a natural next step is to define the constraint-value function over worst-case or quantile rollouts under stochastic disturbances, which would provide a safety guarantee under aleatoric noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes DGPPO, a framework that jointly learns a discrete graph control barrier function (DGCBF) and a distributed safe policy for multi-agent systems with unknown discrete-time dynamics, partial observability, changing neighborhoods, and input constraints. The central theoretical construction (Theorem 2) shows that for a fixed deterministic policy mu, the constraint-value function V^{h,mu}(x) = max_{k>=0} h(x_k) satisfies the DCBF inequality. The algorithm learns this value function via deterministic T-step rollouts with a GNN, updates the policy with a PPO-style loss augmented by CRPO-style safety gradients and an approximate gradient projection (Eqs. (14), (20)), and relies on attention mechanisms for neighborhood changes. Experiments cover LiDAR, MuJoCo, and VMAS environments, comparing against InforMARL and MAPPO-Lagrangian baselines, plus ablations on alpha, nu, rollout type, and learned versus hand-crafted CBFs, and scaling tests up to 512 agents.

Significance. If the results hold, the paper makes a meaningful step for safe multi-agent RL: it removes the need for a known nominal policy and known dynamics, gives a principled value-function construction of discrete CBFs, and demonstrates strong empirical performance across several simulation engines with one hyperparameter set. The proof of Theorem 2 is concise and appears correct, the experimental comparison is broad, and code is provided. The main weakness is that the theoretical safety certificate is not verified for the policy that is actually trained and deployed; the paper's own Limitations paragraph concedes that satisfying the DGCBF condition at all states through learning is hard. The empirical claims are well supported, but the safety guarantee is conditional on a condition that the algorithm never checks.

major comments (4)
  1. [§4.5, Eq. (19)–(20); Theorem 2] Theorem 2 establishes that the exact constraint-value function V^{h,mu} of a fixed policy mu satisfies the DCBF inequality. In DGPPO, V^{h,mu} is learned by regression from finite T-step deterministic rollouts and then inserted into the pseudo-advantage (20), after which mu is updated by the PPO loss (14). Nothing in the training loop re-evaluates the DGCBF condition (16) for the updated policy or certifies the learned V over the state space; the authors' Limitations paragraph concedes that 'achieving this in practice using learning is hard.' Since the learned V can underestimate the true constraint value, the theoretical safety guarantee does not transfer to the deployed policy. The paper should either add a post-training verification step for condition (16), restrict the safety claims to the empirical evaluation, or provide a bound on value-function approximation error that is propagated through policy updates.
  2. [§4.3, Informal Theorem 3 / Theorem A2] The claimed approximate gradient projection g is only orthogonal to the constraint gradients sigma^(m) under the assumption that policy-parameter gradients at different states are mutually orthogonal. This assumption is not satisfied by the graph transformer/GNN policies used in the experiments, which share parameters across all states and agents; the authors state in Section 4.3 that they 'cannot guarantee that the conditions of Informal Theorem 3 hold.' When the assumption fails, the indicator-based modification of the Q-function is not a projection, and the clean decoupling of cost-gradient and safety-gradient updates asserted in Eq. (14) is not established. Given that the ablation in Appendix C.6.1 attributes DGPPO's advantage to this decoupling, this gap is load-bearing and needs either a relaxation of the assumption or a demonstration that the projection error is small.
  3. [§4.4, Informal Theorem 4 / Theorem A3] Theorem A3's sufficient condition for handling neighborhood changes requires the attention weight w(o_ij) to be exactly zero for all agents at distance at least R - 2*bar_d, and additionally assumes that for every transition with a neighborhood change there exists a companion transition with unchanged neighborhood in which the entering/leaving agents are moved outside the sensing radius and all remaining agents move identically. Neither condition is verified for the learned GNN DGCBF: the text says the attention mechanism 'naturally learns' zero weights, but no experiment or structural constraint enforces exact zeros, and the companion-transition condition is not checked. Therefore the validity of (16) under arbitrary neighborhood changes is not guaranteed for the learned DGCBF, which weakens the theoretical contribution for the varying-neighborhood setting.
  4. [§4.3, Theorem A1] Theorem A1 shows that E[max{0,C}] <= 0 implies C <= 0 almost surely under the state-action distribution used in the expectation. This is weaker than the pointwise condition (9b)/(16) required for safety: states not visited by the current rollout distribution are not covered. Since DGPPO uses a stochastic policy during training and a deterministic policy for the learned constraint-value function and for deployment, the almost-sure statement under the stochastic rollout distribution does not directly imply safety under the deterministic closed-loop system. The level of quantification should be stated precisely, and the safety claim should be formulated accordingly.
minor comments (4)
  1. [Appendix A.6, Theorem A4 proof] In the proof of Theorem A4, indices i1 and i2 are defined as arg min_i tilde_B(O_i(x)), while B is defined as max_i tilde_B(O_i(x)); equations (54)–(55) are inconsistent and should use arg max (or the definition of B should be changed consistently).
  2. [General] There are several typos and wording errors: 'it it enters' in Section 3.1, 'we usee' in Appendices C.2.2 and C.2.3, 'asumption' in Appendix A.5, and 'convergences' in Appendix C.6.4; these should be corrected.
  3. [Section 5.3, Figure 6a] The a = 1.2 case violates the condition alpha(-r) > -r for the extended class-kappa function, so its unsafe behavior is a sanity check of the theory rather than evidence about hyperparameter robustness; the text should make this distinction explicit.
  4. [Appendix C.8, Table 3] The generalizability results in Table 3 are reported at matching agent density; adding the average neighbor count or a density statistic would clarify what changes when the number of agents is scaled from 8 to 512.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DCBF construction is a proved identity, and the empirical claims are benchmarked against external baselines.

full rationale

The central theoretical step, Theorem 2, does not reduce to a fitted input: V^{h,mu} is defined as the maximal constraint violation along mu's trajectory, and the proof that it satisfies the DCBF inequality uses the Bellman identity V(x) = max{h(x), V(f(x, mu(x)))}, so the safety condition is a consequence of the definition, not an assumed conclusion. The DGCBF extension (Corollary 1 and Theorem A3) is proved from stated assumptions; the attention-based zero-weight condition is presented as a sufficient condition, and any reliance on the authors' prior GCBF work is an empirical heuristic rather than a substitute for the proof. The empirical main results are measured against external baselines (InforMARL, MAPPO-Lagrangian) on independently specified environments, so no fitted quantity is relabeled as a prediction. The paper's own limitations, namely that the gradient projection conditions of Informal Theorem 3 are not guaranteed for the GNN policies actually used and that learning cannot ensure the DGCBF constraints hold at all states, are explicit correctness gaps rather than instances of deriving a conclusion from its own premise. No load-bearing self-citation chain or construction-level circularity was found.

Assumptions & free parameters 2 free parameters · 9 assumptions · 0 invented entities

The framework adds no new physical entities. It introduces the DGCBF as a mathematical construction and leans on standard RL and CBF results. The main unearned assumptions are the decoupled-parameter orthogonality and the zero-attention-weight conditions, both flagged by the authors as not guaranteed.

free parameters (2)
  • alpha slope a in class-kappa function alpha(r) = a*r = 0.3
    Central hyperparameter for the DGCBF condition; sensitivity analysis in Section 5.3 shows values in (0,1) work but the extreme values fail. It is part of the fixed hyperparameter set used across all environments.
  • nu (weight on the safety-constraint gradient) = 1, doubled at 50% and 75% of training updates
    Schedules the safety/cost tradeoff in the policy loss; sensitivity analysis in Section 5.3 and Appendix C.6.4 shows static choices trade off convergence speed and safety.
assumptions (9)
  • standard math Definition of DCBF and Theorem 1 from Ahmadi et al. (2019) are taken as given.
    Invoked in Section 3.2 to define the discrete barrier condition and the invariance property.
  • standard math Policy gradient theorem and score function identity (Williams 1992) are used for gradient estimation under unknown dynamics.
    Used in Section 4.3 and Appendix A.4 to compute gradients of the DCBF constraint without knowing f.
  • standard math GAE and PPO clipped objective from Schulman et al. (2015, 2017) are used as the base RL algorithm.
    The DGPPO loss in Section 4.5 is built on these standard components.
  • domain assumption Dynamics f are unknown but the environment is sampleable, and agents only observe the local neighborhood within sensing radius R (Equation 2).
    The entire problem setting in Section 3.1 is defined by this observability and unknown-dynamics assumption.
  • domain assumption There exists a maximum travel distance per timestep d_bar and a maximum packing N_bar of agents within radius 2R+2d_bar (Appendix A.7).
    Needed for Lemma 1, Lemma 2, and Theorem A5, which establish that the same DGCBF generalizes to arbitrary N.
  • domain assumption For the generalizability theorem, per-agent dynamics are decoupled: x^{k+1}_i = f0(x^k, u^k) (Equation 63).
    Required for Theorem A5; not satisfied by all tested environments (e.g., the coupled MuJoCo tasks), though those are not part of the generalizability experiment.
  • domain assumption The constraint-value function V^{h,mu}_i can be represented using only local observations o_i (Equation 18, Corollary 1).
    Needed to turn the central value function into a distributed DGCBF; the paper relies on a GNN with attention to achieve this in practice.
  • ad hoc to paper Policy parameter gradients at different states are mutually orthogonal (Theorem A2).
    This is the formal condition behind Informal Theorem 3; the authors explicitly state that this cannot be guaranteed for their GNN policies.
  • ad hoc to paper The attention mechanism learns zero weights for agents at distance >= R-2d_bar, making the learned DGCBF satisfy (16) under neighborhood changes (Theorem A3).
    This is a heuristic borrowed from GCBF+ (Zhang et al. 2025); the learned attention is not verified to satisfy the zero-weight condition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control." pith.science (2026). https://pith.science/paper/FISYKRAP

@misc{pith2026250203640,
  author       = {Pith},
  title        = {Pith review of: Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FISYKRAP}},
  note         = {Machine review of arXiv:2502.03640}
}
read the original abstract

Control policies that can achieve high task performance and satisfy safety constraints are desirable for any system, including multi-agent systems (MAS). One promising technique for ensuring the safety of MAS is distributed control barrier functions (CBF). However, it is difficult to design distributed CBF-based policies for MAS that can tackle unknown discrete-time dynamics, partial observability, changing neighborhoods, and input constraints, especially when a distributed high-performance nominal policy that can achieve the task is unavailable. To tackle these challenges, we propose DGPPO, a new framework that simultaneously learns both a discrete graph CBF which handles neighborhood changes and input constraints, and a distributed high-performance safe policy for MAS with unknown discrete-time dynamics. We empirically validate our claims on a suite of multi-agent tasks spanning three different simulation engines. The results suggest that, compared with existing methods, our DGPPO framework obtains policies that achieve high task performance (matching baselines that ignore the safety constraints), and high safety rates (matching the most conservative baselines), with a constant set of hyperparameters across all environments.

Figures

Figures reproduced from arXiv: 2502.03640 by the authors.

Figure 1
Figure 1. DGPPO algorithm. In addition to the normal MAPPO path (top) using stochastic rollouts, we introduce a second path (bottom) that uses deterministic rollouts to learn a DGCBF. We encourage B˜ to satisfy (16) during neighborhood changes by parameterizing the value functions using graph neural networks (GNN) with graph attention (Velickovi ˇ c et al., 2017), which takes the ´ form of (17) and hence is amenable to Inform… view at source ↗
Figure 2
Figure 2. Environments. We test on (top) LiDAR, (bottom) MuJoCo, and VMAS environments. 5 EXPERIMENTS In this section, we design experiments to answer the following research questions: (Q1) Does DGPPO learn a safe policy that also achieves low costs without hyperparameter tuning in different environments? (Q2) How stable is the training of DGPPO? (Q3) Can DGPPO maintain its perfor￾mance with an increasing number of agents? (Q… view at source ↗
Figure 3
Figure 3. Comparison on N = 3 agents. denotes the mean ± standard deviation. Methods closer to the top left yield lower costs and higher safety rates. 0.0 0.5 1.0 1.5 Step 1e5 0.0 0.5 1.0 Cost Spread 0.0 0.5 1.0 1.5 Step 1e5 0.0 0.5 1.0 Safety rate Spread 0.0 0.5 1.0 1.5 Step 1e5 0.0 0.5 1.0 Cost Line 0.0 0.5 1.0 1.5 Step 1e5 0.6 0.8 1.0 Safety rate Line DGPPO (ours) Schedule Lagr(lr) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Training stability. DGPPO yields smoother training curves compared to the baselines. 0.0 0.5 1.0 1.5 Step 1e5 0.00 0.25 0.50 0.75 1.00 Cost N = 5 0.0 0.5 1.0 1.5 Step 1e5 0.2 0.4 0.6 0.8 1.0 Safety rate N = 5 0.0 0.5 1.0 1.5 Step 1e5 0.00 0.25 0.50 0.75 1.00 Cost N = 7…
Figure 5
Figure 5. Figure 5: Scaling to N = 5, 7. Unlike other methods, DGPPO performs similarly with more agents. hyperparameters result in either focusing only on safety or focusing only on performance. Even with a fixed hyperparameter, the performance of these two baselines also varies between …
Figure 6
Figure 6. Figure 6: Ablations. We vary hyperparameters (top) and verify our design decisions (bottom). 5.3 ABLATION STUDIES We now study hyperparameter sensitivity (Q4) by varying different hyperparameters in DGPPO. Class-κ function α. For the class-κ function in (16), we use a linear α(r…
Figure 7
Figure 7. Figure 7: The results show that using (98) converges much slower in cost, which matches the [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 7
Figure 7. Figure 7: In comparison to DGPPO which uses (100), using (98) converge much slower in cost, [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Costs and safety rates of DGPPO and the baselines during training. [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Comparison between DGPPO with the decoupling method and the CRPO-style update [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: Costs and safety rates of DGPPO and three best baselines in the W [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Comparison between DGPPO with doing constrained optimization without a CBF. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Influence of different ν on the convergence speed and the converged result of DGPPO. The dashed lines show the mean of DGPPO and the baseline with the best performance (Schedule), and the shades show the standard deviation. 0 5 Step 1e3 0 200 400 Cost HalfCheetah 2x3 …
Figure 13
Figure 13. Figure 13: Cost and safety rates of DGPPO and the baselines during training in the Safe multi-agent [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: Costs and safety rates of the converged policies of DGPPO and the baselines in [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Comparison of the converged policies learned using DGPPO and GCBF+ (Zhang et al., [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 57 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pp.\ 22--31. PMLR, 2017

    work page 2017

  2. [2]

    Safe policy synthesis in multi-agent pomdps via discrete-time barrier functions

    Mohamadreza Ahmadi, Andrew Singletary, Joel W Burdick, and Aaron D Ames. Safe policy synthesis in multi-agent pomdps via discrete-time barrier functions. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp.\ 4797--4803. IEEE, 2019

    work page 2019

  3. [3]

    Control barrier function based quadratic programs for safety critical systems

    Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62 0 (8): 0 3861--3876, 2017

    work page 2017

  4. [4]

    Vmas: A vectorized multi-agent simulator for collective robot learning

    Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems, 2022

    work page 2022

  5. [5]

    Benchmarl: Benchmarking multi-agent reinforcement learning

    Matteo Bettini, Amanda Prorok, and Vincent Moens. Benchmarl: Benchmarking multi-agent reinforcement learning. Journal of Machine Learning Research, 25 0 (217): 0 1--10, 2024

    work page 2024

  6. [6]

    Powergridworld: A framework for multi-agent reinforcement learning in power systems

    David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rohit Chintala, Jennifer King, and Ahmed S Zamzam. Powergridworld: A framework for multi-agent reinforcement learning in power systems. In Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, pp.\ 565--570, 2022

    work page 2022

  7. [7]

    Adaptation for validation of consolidated control barrier functions

    Mitchell Black and Dimitra Panagou. Adaptation for validation of consolidated control barrier functions. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp.\ 751--757, 2023. doi:10.1109/CDC49753.2023.10383597

    arXiv 2023

  8. [8]

    An actor-critic algorithm for constrained markov decision processes

    Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & Control Letters, 54 0 (3): 0 207--213, 2005

    work page 2005

Show all 75 references
  1. [9]

    Control barrier certificates for safe swarm behavior

    Urs Borrmann, Li Wang, Aaron D Ames, and Magnus Egerstedt. Control barrier certificates for safe swarm behavior. IFAC-PapersOnLine, 48 0 (27): 0 68--73, 2015

    2015

  2. [10]

    JAX : composable transformations of P ython+ N um P y programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/...

    2018

  3. [11]

    Socially aware motion planning with deep reinforcement learning

    Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 1343--1350. IEEE, 2017 a

    2017

  4. [12]

    Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning

    Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P How. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 285--292. IEEE, 2017 b

    2017

  5. [13]

    Backup control barrier functions: Formulation and comparative study

    Yuxiao Chen, Mrdjan Jankovic, Mario Santillo, and Aaron D Ames. Backup control barrier functions: Formulation and comparative study. In 2021 60th IEEE Conference on Decision and Control (CDC), pp.\ 6835--6841. IEEE, 2021

    2021

  6. [14]

    End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks

    Richard Cheng, G \'a bor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 3387--3395, 2019

    2019

  7. [15]

    Safe nonlinear control using robust neural lyapunov-barrier functions

    Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Conference on Robot Learning, pp.\ 1724--1735. PMLR, 2022

    2022

  8. [16]

    Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning

    Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning. In Learning for Dynamics and Control Conference, pp.\ 315--332. PMLR, 2023

    2023

  9. [17]

    Safe reinforcement learning using robust control barrier functions

    Yousef Emam, Gennaro Notomista, Paul Glotfelter, Zsolt Kira, and Magnus Egerstedt. Safe reinforcement learning using robust control barrier functions. IEEE Robotics and Automation Letters, 2022

    2022

  10. [18]

    Motion planning among dynamic, decision-making agents with deep reinforcement learning

    Michael Everett, Yu Fan Chen, and Jonathan P How. Motion planning among dynamic, decision-making agents with deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 3052--3059. IEEE, 2018

    2018

  11. [19]

    Iterative reachability estimation for safe reinforcement learning

    Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    Learning safe control for multi-robot systems: Methods, verification, and open challenges

    Kunal Garg, Songyuan Zhang, Oswin So, Charles Dawson, and Chuchu Fan. Learning safe control for multi-robot systems: Methods, verification, and open challenges. Annual Reviews in Control, 57: 0 100948, 2024

    2024

  13. [21]

    A reinforcement learning framework for vehicular network routing under peak and average constraints

    Nan Geng, Qinbo Bai, Chenyi Liu, Tian Lan, Vaneet Aggarwal, Yuan Yang, and Mingwei Xu. A reinforcement learning framework for vehicular network routing under peak and average constraints. IEEE Transactions on Vehicular Technology, 2023

    2023

  14. [22]

    Nonsmooth barrier functions with applications to multi-robot systems

    Paul Glotfelter, Jorge Cort \'e s, and Magnus Egerstedt. Nonsmooth barrier functions with applications to multi-robot systems. IEEE control systems letters, 1 0 (2): 0 310--315, 2017

    2017

  15. [23]

    Deadlock analysis and resolution for multi-robot systems

    Jaskaran Singh Grover, Changliu Liu, and Katia Sycara. Deadlock analysis and resolution for multi-robot systems. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pp.\ 294--312. Springer, 2021

    2021

  16. [24]

    Multi-agent constrained policy optimisation

    Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arXiv:2110.02793, 2021

    2021 arXiv

  17. [25]

    Safe multi-agent reinforcement learning for multi-robot control

    Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319: 0 103905, 2023

    2023

  18. [26]

    Optimal control barrier functions for rl based safe powertrain control

    Habtamu Hailemichael, Beshah Ayalew, and Andrej Ivanco. Optimal control barrier functions for rl based safe powertrain control. IFAC-PapersOnLine, 56 0 (3): 0 385--390, 2023

    2023

  19. [27]

    Autocost: Evolving intrinsic cost for zero-violation reinforcement learning

    Tairan He, Weiye Zhao, and Changliu Liu. Autocost: Evolving intrinsic cost for zero-violation reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 14847--14855, 2023

    2023

  20. [28]

    The safety filter: A unified view of safety-critical control in autonomous systems

    Kai-Chieh Hsu, Haimin Hu, and Jaime F Fisac. The safety filter: A unified view of safety-critical control in autonomous systems. Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023

    2023

  21. [29]

    Safedreamer: Safe reinforcement learning with world models

    Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tsE5HLYtYg

    2024

  22. [30]

    Multiagent systems with cbf-based controllers: Collision avoidance and liveness from instability

    Mrdjan Jankovic, Mario Santillo, and Yan Wang. Multiagent systems with cbf-based controllers: Collision avoidance and liveness from instability. IEEE Transactions on Control Systems Technology, 2023

    2023

  23. [31]

    Distributed optimization in multi-agent robotics for industry 4.0 warehouses

    Ajay Kattepur, Hemant Kumar Rath, Anantha Simha, and Arijit Mukherjee. Distributed optimization in multi-agent robotics for industry 4.0 warehouses. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, pp.\ 808--815, 2018

    2018

  24. [32]

    Lidar-based online control barrier function synthesis for safe navigation in unknown environments

    Shaghayegh Keyumarsi, Made Widhi Surya Atman, and Azwirman Gusrialdi. Lidar-based online control barrier function synthesis for safe navigation in unknown environments. IEEE Robotics and Automation Letters, 2023

    2023

  25. [33]

    Rpcbf: Constructing safety filters robust to model error and disturbances via policy control barrier functions

    Luzia Knoedler, Oswin So, Ji Yin, Mitchell Black, Zachary Serlin, Panagiotis Tsiotras, Javier Alonso-Mora, and Chuchu Fan. Rpcbf: Constructing safety filters robust to model error and disturbances via policy control barrier functions. arXiv preprint, 2024

    2024

  26. [34]

    Control barrier functions for multi-agent systems under conflicting local signal temporal logic tasks

    Lars Lindemann and Dimos V Dimarogonas. Control barrier functions for multi-agent systems under conflicting local signal temporal logic tasks. IEEE control systems letters, 3 0 (3): 0 757--762, 2019

    2019

  27. [35]

    Learning hybrid control barrier functions from data

    Lars Lindemann, Haimin Hu, Alexander Robey, Hanwen Zhang, Dimos Dimarogonas, Stephen Tu, and Nikolai Matni. Learning hybrid control barrier functions from data. In Conference on robot learning, pp.\ 1351--1370. PMLR, 2021

    2021

  28. [36]

    Conflict-averse gradient descent for multi-task learning

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 0 18878--18890, 2021 a

    2021

  29. [37]

    Cmix: Deep multi-agent reinforcement learning with peak and average constraints

    Chenyi Liu, Nan Geng, Vaneet Aggarwal, Tian Lan, Yuan Yang, and Mingwei Xu. Cmix: Deep multi-agent reinforcement learning with peak and average constraints. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, S...

    2021

  30. [38]

    Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning

    Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 6252--6259. IEEE, 2018

    2018

  31. [39]

    Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning

    Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Ba s ar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 8767--8775, 2021

    2021

  32. [40]

    Safe value functions

    Pierre-François Massiani, Steve Heim, Friedrich Solowjow, and Sebastian Trimpe. Safe value functions. IEEE Transactions on Automatic Control, 68 0 (5): 0 2743--2757, 2023

    2023

  33. [41]

    Scalable multi-agent reinforcement learning through intelligent information aggregation

    Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, and Hamsa Balakrishnan. Scalable multi-agent reinforcement learning through intelligent information aggregation. In International Conference on Machine Learning, pp.\ 25817--25833. PMLR, 2023

    2023

  34. [42]

    Safe optimal control using stochastic barrier functions and deep forward-backward sdes

    Marcus Pereira, Ziyi Wang, Ioannis Exarchos, and Evangelos Theodorou. Safe optimal control using stochastic barrier functions and deep forward-backward sdes. In Conference on Robot Learning, pp.\ 1783--1801. PMLR, 2021

    2021

  35. [43]

    Decentralized safe multi-agent stochastic optimal control using deep fbsdes and admm

    Marcus A Pereira, Augustinos D Saravanos, Oswin So, and Evangelos A Theodorou. Decentralized safe multi-agent stochastic optimal control using deep fbsdes and admm. arXiv preprint arXiv:2202.10658, 2022

    2022 arXiv

  36. [44]

    Automated and formal synthesis of neural barrier certificates for dynamical models

    Andrea Peruffo, Daniele Ahmed, and Alessandro Abate. Automated and formal synthesis of neural barrier certificates for dynamical models. In International conference on tools and algorithms for the construction and analysis of systems, pp.\ 370--388. Springer, 2021

    2021

  37. [45]

    Learning safe multi-agent control with decentralized neural barrier certificates

    Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi-agent control with decentralized neural barrier certificates. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=P6_q1BRxY8Q

    2021

  38. [46]

    Control barrier function-based quadratic programs introduce undesirable asymptotically stable equilibria

    Matheus F Reis, A Pedro Aguiar, and Paulo Tabuada. Control barrier function-based quadratic programs introduce undesirable asymptotically stable equilibria. IEEE Control Systems Letters, 5 0 (2): 0 731--736, 2020

    2020

  39. [47]

    Learning barrier functions for constrained motion planning with dynamical systems

    Matteo Saveriano and Dongheui Lee. Learning barrier functions for constrained motion planning with dynamical systems. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 112--119. IEEE, 2019

    2019

  40. [48]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  41. [49]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  42. [50]

    Multi-agent motion planning for dense and dynamic environments via deep reinforcement learning

    Samaneh Hosseini Semnani, Hugh Liu, Michael Everett, Anton De Ruiter, and Jonathan P How. Multi-agent motion planning for dense and dynamic environments via deep reinforcement learning. IEEE Robotics and Automation Letters, 5 0 (2): 0 3221--3226, 2020

    2020

  43. [51]

    Safe, multi-agent, reinforcement learning for autonomous driving

    Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016

    2016 arXiv

  44. [52]

    Masked label prediction: Unified message passing model for semi-supervised classification

    Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509, 2020

    2009 arXiv

  45. [53]

    Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning

    Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. In Proceedings of Robotics: Science and Systems, 2023

    2023

  46. [54]

    How to train your neural control barrier function: Learning safety filters for complex input-constrained systems

    Oswin So, Zachary Serlin, Makai Mann, Jake Gonzales, Kwesi Rutledge, Nicholas Roy, and Chuchu Fan. How to train your neural control barrier function: Learning safety filters for complex input-constrained systems. In 2024 IEEE International Conference on Robotics and Automation...

    2024

  47. [55]

    Synthesis of control barrier functions using a supervised machine learning approach

    Mohit Srinivasan, Amogh Dabholkar, Samuel Coogan, and Patricio A Vela. Synthesis of control barrier functions using a supervised machine learning approach. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 7139--7145. IEEE, 2020

    2020

  48. [56]

    A predictive safety filter for learning-based racing control

    Ben Tearle, Kim P Wabersich, Andrea Carron, and Melanie N Zeilinger. A predictive safety filter for learning-based racing control. IEEE Robotics and Automation Letters, 6 0 (4): 0 7635--7642, 2021

    2021

  49. [57]

    Mankowitz, and Shie Mannor

    Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019

    2019

  50. [58]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012. doi:10.1109/IROS.2012.6386109

    2012

  51. [59]

    Graph attention networks

    Petar Veli c kovi \'c , Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017

    2017 arXiv

  52. [60]

    Safety barrier certificates for collisions-free multirobot systems

    Li Wang, Aaron D Ames, and Magnus Egerstedt. Safety barrier certificates for collisions-free multirobot systems. IEEE Transactions on Robotics, 33 0 (3): 0 661--674, 2017

    2017

  53. [61]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  54. [62]

    Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks

    Tong Wu, Pan Zhou, Kai Liu, Yali Yuan, Xiumin Wang, Huawei Huang, and Dapeng Oliver Wu. Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Transactions on Vehicular Technology, 69 0 (8): 0 8243--8256, 2020

    2020

  55. [63]

    Control barrier functions for systems with high relative degree

    Wei Xiao and Calin Belta. Control barrier functions for systems with high relative degree. In 2019 IEEE 58th conference on decision and control (CDC), pp.\ 474--479. IEEE, 2019

    2019

  56. [64]

    Crpo: A new approach for safe reinforcement learning with convergence guarantee

    Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp.\ 11480--11491. PMLR, 2021

    2021

  57. [65]

    Robustness of control barrier functions for safety critical control

    Xiangru Xu, Paulo Tabuada, Jessy W Grizzle, and Aaron D Ames. Robustness of control barrier functions for safety critical control. IFAC-PapersOnLine, 48 0 (27): 0 54--61, 2015

    2015

  58. [66]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 0 24611--24624, 2022

    2022

  59. [67]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 0 5824--5836, 2020

    2020

  60. [68]

    Safe reinforcement learning using robust mpc

    Mario Zanon and S \'e bastien Gros. Safe reinforcement learning using robust mpc. IEEE Transactions on Automatic Control, 66 0 (8): 0 3638--3652, 2020

    2020

  61. [69]

    Neural graph control barrier functions guided distributed collision-avoidance multi-agent control

    Songyuan Zhang, Kunal Garg, and Chuchu Fan. Neural graph control barrier functions guided distributed collision-avoidance multi-agent control. In Conference on Robot Learning, pp.\ 2373--2392. PMLR, 2023

    2023

  62. [70]

    Gcbf+: A neural graph control barrier function framework for distributed safe multi-agent control

    Songyuan Zhang, Oswin So, Kunal Garg, and Chuchu Fan. Gcbf+: A neural graph control barrier function framework for distributed safe multi-agent control. IEEE Transactions on Robotics, 2025

    2025

  63. [71]

    Multi-agent first order constrained optimization in policy space

    Youpeng Zhao, Yaodong Yang, Zhenbo Lu, Wengang Zhou, and Houqiang Li. Multi-agent first order constrained optimization in policy space. Advances in Neural Information Processing Systems, 36, 2024

    2024

  64. [72]

    Decentralized safe and scalable multi-agent control under limited actuation

    Vrushabh Zinage, Abhishek Jha, Rohan Chandra, and Efstathios Bakolas. Decentralized safe and scalable multi-agent control under limited actuation. arXiv preprint arXiv:2409.09573, 2024

    2024 arXiv

  65. [73]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  66. [74]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  67. [75]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.