SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning

Hangguan Shan; Na Li; Wei Ni; Wenjie Zhang; Xinyu Li

REVIEW 4 major objections 5 minor 17 references

SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that SHAP state attributions, computed from a learned value function, can be piped directly into a kernelized actor-critic, yielding a two-timescale algorithm that provably converges under state perturbations.

desk verdict Good idea and honest experiments, but the flagship convergence theorem analyzes a different update than the algorithm actually runs. read the letter →

arxiv 2512.05291 v3 pith:UDS5CUAE submitted 2025-12-04 cs.LG

Na Li , Hangguan Shan , Wei Ni , Wenjie Zhang , Xinyu Li This is my paper · ORCID

classification cs.LG

keywords ExplainablereinforcementlearningShapleyvalues/SHAPReproducingkernelHilbertspacesActor-criticmethodsStateperturbationsNon-asymptoticconvergenceContinuouscontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RSA2C puts the Actor, Value Critic, and Advantage Critic in kernel-based function spaces (RKHS) and uses SHAP attributions—computed analytically from the Value Critic via kernel mean embeddings—to rescale state features inside an adaptive Mahalanobis kernel. The point is to make interpretability intrinsic to training: the same attribution scores that explain decisions also shape policy gradients and advantage targets. The paper's central claim is that this attribution-aware mechanism does not compromise learning theory: it derives a global non-asymptotic convergence bound under state perturbations, with the performance gap decomposing into a perturbation-error term and a convergence-error term. If the claim holds, explainable RL no longer has to be a post-hoc afterthought—explanations can be a load-bearing part of the optimization loop, and a non-neural kernel method can match deep RL baselines on low-dimensional continuous control while staying substantially cheaper.

What carries the argument

The central object is the adaptive Mahalanobis-weighted operator-valued kernel K(s,s_j) = κ_φ(s,s_j) Σ_K, where κ_φ is a Gaussian kernel whose per-dimension length scales are set by SHAP attributions (floored at a small positive constant). This kernel defines the policy mean h(s) in a vector-valued RKHS and is shared by the Advantage Critic; the SHAP scores themselves come from the scalar-RKHS Value Critic through kernel mean/conditional mean embeddings. The theoretical machinery that carries the convergence proof is the compatible function approximation assumption: the Advantage Critic's feature ν(s,a) must equal the policy score ∇_h log π(a|s) = K(s,·)Σ^{-1}(a−h(s)), so that the critic can

What would settle it

On the LQR benchmark with a known optimal value function, compute the empirical policy-gradient estimate used by the actual implementation and compare it to the true policy gradient; if the difference does not shrink as the dictionary grows, the compatible-function premise is violated. Alternatively, inspect the Advantage Critic feature in the released code: if it uses Σ^{-1/2} as written in Section 3.1, the convergence theorem in Appendix C.5 does not apply to the implemented algorithm.

Watch

Extended reading notes

Core claim

The paper claims to establish a global, non-asymptotic convergence bound for a two-timescale RKHS actor-critic with SHAP-guided feature weighting under state perturbations. The algorithm estimates state attributions from the Value Critic using two routes—kernel mean embeddings for on-manifold expectations and conditional mean embeddings for off-manifold expectations—then converts them into Mahalanobis weights that modulate both the Actor's kernel and the Advantage Critic's targets. The theory shows the expected optimality gap is bounded by an approximation error, a perturbation term that vanishes as the perturbation magnitude goes to zero, and a convergence term of order O(log²T / T^{1/4}) a

Load-bearing premise

The guarantee rests on the Advantage Critic using exactly the same feature as the policy's score gradient; the paper's Section 3.1 defines this feature with a half-inverse covariance factor while the Appendix B.3 proof uses the full inverse, so the exact matching condition—and with it the unbiased-gradient argument—is not established for the algorithm as described.

Editorial extensions

If this is right

If correct, explainability becomes intrinsic: the SHAP values shown to a user are the exact signals that reweight the Actor's kernel and Advantage Critic, so the explanation is not a post-hoc overlay.
Kernel-based (non-neural) actor-critics can be competitive with deep RL on low-dimensional continuous control, with orders-of-magnitude lower FLOPs and wall-clock time per update.
The convergence bound gives a concrete stability guarantee under bounded state perturbations: performance degrades gracefully with the perturbation magnitude, rather than abruptly.
Without perturbations, the algorithm is claimed to reach ε error with O((1−γ)^{-5} ε^{-4} log²(1/ε)) samples—a finite-time guarantee for a kernel-based actor-critic.
The CME (off-manifold) variant is claimed to be more robust to state noise than the KME variant, because it models feature correlations when imputing missing states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the convergence proof appears to require the Advantage Critic feature to be K(s,·)Σ^{-1}(a−h(s)), but the algorithm definition in Section 3.1 uses K(s,·)Σ^{-1/2}(a−h(s)); a reader or user should verify which feature the released code implements, because the theorem's unbiased-gradient premise holds only for the first form.
Editorial inference: the same Mahalanobis weighting scheme could be lifted to off-policy or deep actor-critics by using kernel approximations such as random Fourier features; the paper's own conclusion gestures in this direction for pixel-based observations.
Editorial inference: the perturbation bound assumes a deterministic, time-invariant adversary; stochastic or adaptive adversaries—common in robust control—would be a natural stress test and may require a different proof technique.
Editorial inference: if the compatible-function gap is resolved, an immediate testable prediction is that attribution-weighted updates should outperform unweighted kernel actor-critic exactly in regimes where state dimensions are heterogeneous in reward relevance; the paper's ablation study already shows this on Pendulum and BipedalWalker.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper pairs RKHS-SHAP attributions with a kernelized two-timescale actor-critic, and that combination is genuinely new relative to prior work (Chau et al. is post-hoc, Lever and Stafford and Yang et al. don't use SHAP-style weights). The empirical part is careful: ten seeds, FLOPs and runtime reported, noise-robustness tables, and the Ant-v5 saturation is disclosed rather than hidden. Code is available. For the algorithmic idea, the ablation on Pendulum and BipedalWalker is plausible and the CME variant does look more stable under noise. Unfortunately, the theory doesn't hold together. The most serious problem is that Theorem 2's proof analyzes score ascent, not RSA2C. Algorithm 1 / Eq. (18) defines the actor update as h_{t+1} = h_t + alpha_t^h * (1/n) sum_i A-hat(s_i,a_i) K(s_i,.) Sigma^{-1}(a_i-h_t(s_i)) - advantage-weighted. But in Appendix C.5 the unprojected step is written as h_{t+1}-h_t = alpha_t^h g-hat(h_t), with g-hat called a score estimate. The bias bounds (56)-(57) and the telescoping in Step 3 treat g-hat as an unbiased estimator of E[g], not of E[A-hat g]. Those are different unless A-hat is identically 1. So the non-asymptotic bound applies to a different algorithm - pure score ascent on D(h) = E_{nu^{pi*}}[log pi_h] - not to RSA2C. Theorem 3 inherits this. That's a load-bearing flaw. Two smaller issues: Proposition 1's proof (C.3) simply states 'we assume a coercivity condition' that is exactly what the proposition claims; there's no derivation. And the Advantage Critic feature is defined with Sigma^{-1/2} in Section 3.1 but the compatibility proof in B.3 uses Sigma^{-1}. If the algorithm uses Sigma^{-1/2}, the unbiased-gradient step is wrong; if it uses Sigma^{-1}, the definition in Section 3.1 is wrong. Either way, the compatibility step doesn't go through as written. The kernel-drift handling of the self-referential SHAP loop is legitimate and worth keeping. The empirical method may be salvageable with a corrected proof or a changed definition, but the current preprint does not support its main theoretical claims. Who this is for: someone working on kernel RL or explainable RL who wants the algorithmic idea and the empirical comparison - not the theory as it stands. A serious referee should see it, because the flaws are subtle and the empirical direction has value, but the manuscript needs major revision before its guarantees can be taken at face value.

Referee Report

4 major / 5 minor

Summary. The paper proposes RSA2C, a kernelized two-timescale actor-critic algorithm in which SHAP attributions computed from the Value Critic are turned into Mahalanobis weights for the Actor kernel and Advantage Critic targets. The Actor lives in a vector-valued RKHS with an operator-valued kernel; the Value and Advantage Critics live in scalar RKHSs with sparse ALD dictionaries. The main theoretical claim is a global non-asymptotic convergence bound under state perturbations (Theorems 2 and 3), decomposed into a perturbation error and a convergence error. Experiments are reported on Pendulum-v1, BipedalWalker-v3, Ant-v5, and a linear-quadratic benchmark, with code released.

Significance. If established, the convergence guarantee would be a notable first for SHAP-guided, kernel-based actor-critic methods under adversarial state perturbations, and the interpretability-through-training idea is timely. The manuscript has strengths: it ships executable code, reports systematic ablations, compares with deep RL baselines, and includes a dedicated LQR experiment with a closed-form optimal value function. However, the central theoretical result is not supported as written: the proof of Theorem 2 analyzes a score-ascent update and an oracle cross-entropy objective, not the advantage-weighted on-policy update actually implemented in Algorithm 1, and the compatible-feature definition is internally inconsistent. These are load-bearing problems for the main contribution.

major comments (4)

[Appendix C.5 (Theorem 2), Eq. (54); Algorithm 1 / Eq. (18)] The proof of Theorem 2 defines the Actor update as h_{t+1}-h_t = alpha_t^h \hat g(h_t), where \hat g is called a score estimate of g(h)=∇_h log π_h (Eqs. (51)-(54)). But Algorithm 1 line 5, with the concrete form in Eq. (18), updates h_{t+1}=h_t+alpha_t^h (1/n) Σ_i \hat A_{w_A}(s_i,a_i) K(s_i,·)Σ^{-1}(a_i-h_t(s_i)). This estimates E[\hat A g], not E[g]; these coincide only if \hat A ≡ 1. Consequently the identities in Eq. (56)-(57), the bias bound in Eq. (73)-(74), and the telescoped bound in Step 3 prove convergence of a different algorithm. Moreover, D(h)=E_{ν^{π⋆}}[log π_h] is an oracle cross-entropy whose gradient is E_{a∼π⋆}[∇ log π_h], whereas RSA2C samples actions from the on-policy π_h; even a pure score estimator would be biased for this oracle gradient. Theorem 3 inherits the gap, so the paper's central non-asymptotic guarantee is unproven for the proposed method.
[Section 3.1, Eq. (5); Appendix B.3, Eq. (20)] There is an internal inconsistency in the compatible-function-approximation argument. Section 3.1 defines the Advantage Critic feature as ν(s,a)=K(s,·)Σ^{-1/2}(a-h(s)) and the associated scalar kernel uses Σ^{-1/2}. The policy score derived in Proposition 2/Eq. (24) and used in the compatibility proof is ∇_h log π_h(a|s)=K(s,·)Σ^{-1}(a-h(s)). Appendix B.3 writes ∇_{w_A} A = ν(s,a)=K(s,·)Σ^{-1}(a-h(s)). Compatible function approximation requires ν = ∇_h log π_h. If the algorithm genuinely uses Σ^{-1/2}, the gradient estimate is biased and the unbiased-policy-gradient identity used in Theorem 2 collapses; if it uses Σ^{-1}, then Eq. (5) and the kernel definition in §3.1 are inconsistent. Either way, the compatibility condition (20) is not satisfied as stated.
[Appendix C.3, Proposition 1 and Eq. (26)] Proposition 1 claims that the Fisher information operator satisfies F(h) ⪰ λ_F I_m, but the proof consists of assuming Eq. (26): 'We assume a coercivity condition that there exists λ_F > 0 such that ...'. This is circular; the proposition does not derive the coercivity from the stated assumptions. Since this proposition is presented as a formal part of the theoretical guarantees and the coercivity of the Fisher operator is asserted rather than established, the statement should either be proved from the assumptions or removed/reclassified as an assumption.
[Assumption 4 and Appendix C.5, Eq. (65)] The proof repeatedly uses that the score estimator satisfies E∥g(h_t)-\hat g(h_t)∥²_{H_K} ≤ C_b/n_eff (Assumption 4, used in Eqs. (65), (73)-(80)). For RSA2C's actual \hat g from Eq. (18), which is advantage-weighted, this bound has not been verified and in fact the expectation of the estimator is not the score. Thus Assumption 4 is not an innocuous sampling condition; it is effectively part of the conclusion that needs to be proved for the implemented update. The mismatch makes the sample-complexity claim in Theorem 2 unsupported.

minor comments (5)

[Algorithm 1 and Section 3.1] Algorithm 1 line 5 says '∇_hJ(h,Σ) defined in (4)', but Eq. (4) defines the Gaussian policy, not the gradient. The gradient appears later in Eq. (15)/(18); the cross-reference should be fixed.
[Theorem 1] The statement says 'Under Assumptions 2 and 2' twice. This appears to be a typo; one of the two references should point to a different assumption.
[Assumption 5] The final sentence 'Assumption 3 holds with C_ν = O(...)' conflates two different objects: Assumption 3 concerns Gram-matrix invertibility, while C_ν is the Lipschitz constant of the visitation distribution. Please reword.
[Figure 11 caption] The caption reads 'onvergence of the result gap'; missing 'C' in 'Convergence'.
[Appendix A.1] The text says 'Weevaluateonthreecontinuous-controlenvironments...includingPendulum-v1andBipedalWalker-v3.' Ant-v5 is described later but omitted from this list; all three environments should be named.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity in the central derivation; one standalone proposition is assumed into existence, and the main convergence proof has a non-circular correctness gap.

self definitional [Proposition 1 / Appendix C.3 (Proof of Proposition 1)]
"Proposition 1. ... F(h) = E_{ν^{π_h}}[∇_h log π_h(a|s)∇_h log π_h(a|s)^T] ⪰ λ_F · I_m. ... We assume a coercivity condition that there exists a constant λ_F > 0 such that ⟨f, F(h)f⟩_{H_K} ≥ λ_F · ∥f∥^2_{H_K}, ∀f ∈ H_K."

The proof supplies no derivation of the coercivity bound from the Gaussian policy or the RKHS structure; it simply states the existence of λ_F, which is exactly the content of Proposition 1. The proposition is therefore equivalent to its own assumption. It is not invoked in Theorems 1-3, so the circularity is non-load-bearing.

full rationale

The central RSA2C loop — SHAP from the Value Critic modulating the Actor kernel and Advantage Critic targets — is handled in the proof as kernel drift (Lemma 6), with the Value Critic explicitly left unweighted to avoid self-referential value/attribution dependence. That is a legitimate treatment rather than a circular reduction. No fitted parameter is relabeled as a prediction: the LQR test computes an exact Riccati gap and the empirical benchmarks are external. The compatibility argument rests on Sutton et al. (1999), not on a self-citation, and no uniqueness/ansatz is imported from the authors' own prior work. I found one actual circular step: Proposition 1's proof merely assumes the coercivity it claims to prove. It does not feed into the main convergence theorems, so it does not affect the central derivation. Separately, there is a serious non-circular correctness gap in Theorem 2: Eq. (54) defines the update as h_{t+1}-h_t = α_t^h \hat g(h_t) with \hat g a 'score estimate', whereas Algorithm 1 / Eq. (18) use an advantage-weighted update \hat A K(s,·)Σ^{-1}(a-h(s)); Eq. (56) then requires \hat g to be unbiased for g, which is not established and is false for the advantage-weighted estimator (E[\hat A g] ≠ E[g] unless \hat A ≡ 1). Section 3.1 Eq. (5) also uses Σ^{-1/2} while Appendix B.3 uses Σ^{-1}. These are proof gaps, not circularity, and should be scored as correctness risk.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The paper introduces algorithmic constructs (Mahalanobis-weighted OVK, Advantage Critic, SHAP-gated gradients) but no new physical or ontological entities; all are functions of existing quantities. The main ledger items are the strong theoretical assumptions (Assumptions 1-5, Eq. 26) and the hand-chosen per-environment hyperparameters that the empirical claims depend on.

free parameters (6)

RBF kernel variance (length-scale squared) = 0.8 (Pendulum), 3.0 (BipedalWalker), 10.0 (Ant)
Chosen per environment (Tables 6-8); controls smoothness of the kernel and strongly affects empirical performance and the coherence bounds in Assumption 3.
Max dictionary size q = 384 (Pendulum), 1024 (BipedalWalker), 4096 (Ant)
Hand-set budgets for ALD sparsification; they determine approximation capacity and runtime, and interact with Assumption 3 (q-1)ρ<1.
Actor covariance schedule (initial/final) = 0.35I/0.25I (Pendulum), 0.9I/0.4I (BipedalWalker), I/0.3I (Ant)
Exploration variance annealed by hand; changes the Fisher information and policy-gradient scale, and affects the D_TV bound in Theorem 1.
Value Critic learning rate ν = 0.01 (Pendulum), 0.005 (BipedalWalker), 0.05 (Ant)
Per-environment choice meant to satisfy the two-timescale condition α^v/α^h → 0; directly affects the tracking-error rate in Lemma 11.
Actor learning rate α (called σ in tables) = 1.0 (Pendulum), 0.08 (BipedalWalker), 0.25 (Ant)
Per-environment choice for the slow-timescale update; controls the convergence rate in Theorem 2.
SHAP floor ε0
Introduced in Definition 1 to ensure W ≻ 0; the paper does not specify its value or how it is chosen, yet it affects all kernel similarities and thus the Actor update.

assumptions (7)

standard math Standard MDP, discounted return, policy gradient theorem, performance difference lemma (Lemma 10)
Assumed throughout as background; cited via Sutton et al. 1999 and Lemma 10.
domain assumption Assumption 1: perturbation is a deterministic, Markovian, time-invariant b(s) restricted to a set B(s)
Defines the adversarial perturbation model in Section 4; needed to invoke Lemma 1 (Zhang et al. 2020) for the value gap bound.
domain assumption Assumption 2: value-critic feature map is bounded and Lipschitz, kernel is Lipschitz
Required for Lemma 2/3 perturbation bounds on SHAP values and for Lemma 5 gradient growth.
ad hoc to paper Assumption 3: Gram matrices K_V,V and K_A,A are invertible with λ_min ≥ 1-(q-1)ρ (RBF min-separation/diagonal dominance)
A very strong coherence condition; for q=4096 it requires dictionary centers separated so that ρ < 1/4095, which is not verified in the experiments. It underpins Lemmas 3, 4, 8.
domain assumption Assumption 4: on-policy geometric mixing with effective sample size n_eff ≍ n/τ_mix and score-estimate variance bound
Needed for Bernstein-type concentration in Lemma 11; plausible but not verified on the environments.
domain assumption Assumption 5: visitation distribution is C_ν-Lipschitz in h (TV norm), policy is C_π-Lipschitz, Markov chain is uniformly ergodic
Strong smoothness/ergodicity conditions used for Lemma 6 and Lemma 9 drift bounds; no empirical evidence provided.
ad hoc to paper Eq. (26): Fisher information operator F(h) is coercive, ⟨f, F(h)f⟩ ≥ λ_F ||f||²
Proposition 1 states this as a proven result, but Appendix C.3 only says 'We assume a coercivity condition...' — the proposition's conclusion is assumed, not derived.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning." pith.science (2026). https://pith.science/paper/UDS5CUAE

@misc{pith2026251205291,
  author       = {Pith},
  title        = {Pith review of: SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UDS5CUAE}},
  note         = {Machine review of arXiv:2512.05291}
}

read the original abstract

Actor-critic (AC) methods are a cornerstone of reinforcement learning (RL) but offer limited interpretability. Current explainable RL methods seldom use state attributions to assist training. Rather, they treat all state features equally, thereby neglecting the heterogeneous impacts of individual state dimensions on the reward. We propose RKHS-SHAP-based Advanced Actor-Critic (RSA2C), an attribution-aware, kernelized, two-timescale AC algorithm, including Actor, Value Critic, and Advantage Critic. The Actor is instantiated in a vector-valued reproducing kernel Hilbert space (RKHS) with a Mahalanobis-weighted operator-valued kernel, while the Value Critic and Advantage Critic reside in scalar RKHSs. These RKHS-enhanced components use sparsified dictionaries: the Value Critic maintains its own dictionary, while the Actor and Advantage Critic share one. State attributions, computed from the Value Critic via RKHS-SHAP (kernel mean embedding for on-manifold and conditional mean embedding for off-manifold expectations), are converted into Mahalanobis-gated weights that modulate Actor gradients and Advantage Critic targets. We derive a global, non-asymptotic convergence bound under state perturbations, showing stability through the perturbation-error term and efficiency through the convergence-error term. Empirical results on three continuous-control environments show that RSA2C achieves efficiency, stability, and interpretability. Our code is available at https://github.com/Na-Li66/RSA2C.

Figures

Figures reproduced from arXiv: 2512.05291 by the authors.

**Figure 2.** Ablation study. 0 500 1000 1500 2000 Epoch 1400 1200 1000 800 600 400 200 0 Return RSA2C-CME RSA2C-KME SAC PPO [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 4.** Visualization on interpretability of RSA2C on [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Ablation study on BipedalWalker-v3. 0 300 600 900 1200 1500 Epoch 100 0 100 200 300 Return RSA2C-CME RSA2C-KME SAC PPO [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

**Figure 7.** Figure 7: Visualization on interpretability of RSA2C on [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: Ablation study on Ant-v5. 0 400 800 1200 1600 2000 Epoch 1000 0 1000 2000 3000 4000 5000 6000 Return RSA2C-CME RSA2C-KME SAC PPO [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

**Figure 10.** Figure 10: Visualization on interpretability of RSA2C on [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

**Figure 11.** Figure 11: onvergence of the result gap J(π) − J(π ⋆ ) on the LQR environment B Algorithm details B.1 Online Sparsification in RSA2C Unlike parametric models with fixed-size representations, successively updating RSA2C by incrementally adding new components leads to increasingly…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 3 linked inside Pith

[1]

M., Lee, J

Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2020). On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Aissing, G. and Monkhorst, H. J. (1992). Linear dependence in basis-set calculations for extended systems. International journal of quantum chemistry, 43(6):733–745. Alvarez, M. A., Rosasco, L., Lawre...

2020
[2]

Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise.arXiv preprint arXiv:1706.03825. Štrumbelj, E. and Kononenko, I. (2014). Explaining prediction models and individual predictions with feature contributions.Knowledge and information systems, 41:647–665. Sundararajan, M., Taly, A., and Yan, Q...

arXiv 2017
[10]

Sutton, R

MIT press Cambridge. Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems,

1999
[11]

Verma, A., Murali, V., Singh, R., Kohli, P., and Chaudhuri, S. (2018). Programmatically interpretable reinforcement learning. InInternational conference on machine learning, pages 5045–5054. PMLR. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learn- ing.Machine learning, 8(3):229–256. Xu, T., Wang...

arXiv 2018
[12]

and Stafford, R

Lever, G. and Stafford, R. (2015). Modelling policies in mdps in reproducing kernel hilbert space. InArtificial intelligence and statistics, pages 590–598. PMLR. Li, J., Kuang, K., Wang, B., Liu, F., Chen, L., Wu, F., and Xiao, J. (2021). Shapley counterfactual credits for multi-agent reinforcement learning. InProceedings of the 27th ACM SIGKDD Conference...

2015
[13]

Across nonzero noise levels, RSA2C- CME shows markedly lower variability, with standard deviations from 24.75 to 43.06, whereas RSA2C-KME ranges from 46.74 to 56.03. For the mean return, the two variants are similar at light noise 0 and 0.001, and CME outperforms KME at higher noise 0.005 and 0.01, yielding 264.59 versus 242.89 and 254.85 versus 242.05, r...
[15]

And we can estimate∇hJ(h,Σ)by sampling nstate-action pairs{s ι,a ι, rι,s ι+1}and approximating with the average

Note that the steepest ascent direction is a function inH. And we can estimate∇hJ(h,Σ)by sampling nstate-action pairs{s ι,a ι, rι,s ι+1}and approximating with the average. B.3 Compatible function approximation In this section, we demonstrate that the Advantage Critic and Value Critic satisfied the compatible function architecture. To begin with, we have b...

1999
[16]

•L V: Lipschitz drift of the Value Critic w.r.t

σ with0< ν < σ≤1. •L V: Lipschitz drift of the Value Critic w.r.t. actor parameters:∥w⋆ V(h′)−w ⋆ V(h)∥ ≤LV∥h′ −h∥. Whenonlyscalarkernelsareinvolved, onemaysafelyreplaceM K, MΓ byM k. Foroperator-valuedkernels / vector-valued RKHS, keepMK (kernel boundedness) andMΓ (second-order operator bound) separate. C.2 Supporting Lemmas Lemma 1(Theorem 5 in Zhang et...

2020

Show all 17 references

[17]

−2σ , then all implementation-driven terms fall intoSα2 (T)/S α(T)and are strictly dominated by the leading rates in (87). With fixed budgets (constantq t, εPV,t orn eff,t), the corresponding residuals create a floor of order O (1−γ) −1T −σ or constant; the schedule in (83) (o...

2022
[20]

why should i trust you?

Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). " why should i trust you?" explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144. Ribeiro, M. T., Singh, S., and Gu...

2016
[25]

C.(2022)

Yang, Y., Zhu, H., Zhang, Q., Zhao, B., Li, Z., and Wunsch, D. C.(2022). Sparse onlinekernelized actor-critic learning in reproducing kernel hilbert space.Artificial Intelligence Review, 55(1):23–58. Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. ...

2022
[30]

Micchelli, C. A. and Pontil, M. (2005). On learning vector-valued functions.Neural computation, 17(1):177–

2005
[31]

S., Ghavamzadeh, M., and Lee, M

Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., and Lee, M. (2009). Natural actor–critic algorithms. Automatica, 45(11):2471–2482. Cayci, S., He, N., and Srikant, R. (2024). Finite-time analysis of entropy-regularized neural natural actor- critic algorithm.Transactions on Machi...

2009
[32]

Romero, A., Song, Y., and Scaramuzza, D. (2024). Actor-critic model predictive control. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14777–14784. IEEE. Rosynski, M., Kirchner, F., and Valdenegro-Toro, M. (2020). Are gradient-based saliency maps...

2024 arXiv
[204]

Milani, S., Topin, N., Veloso, M., and Fang, F. (2024). Explainable reinforcement learning: A survey and comparative review.ACM Computing Surveys, 56(7):1–36. Muandet, K., Fukumizu, K., Sriperumbudur, B., Schölkopf, B., et al. (2017). Kernel mean embedding of distributions: A ...

2024
[1000]

Figure9showsthatdeepRLalgorithmsareabletodiscoversignificantlybettersolutions in high-dimensional state space

This indicates that RSA2C remains stable but becomes trapped in a local optimum when the state dimensionality is large and the underlying dynamics arehighlynonlinear. Figure9showsthatdeepRLalgorithmsareabletodiscoversignificantlybettersolutions in high-dimensional state space....

1950
[2022]

and Tsitsiklis, J

Konda, V. and Tsitsiklis, J. (1999). Actor-critic algorithms.Advances in neural information processing systems,

1999

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

M., Lee, J

Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. (2020). On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Aissing, G. and Monkhorst, H. J. (1992). Linear dependence in basis-set calculations for extended systems. International journal of quantum chemistry, 43(6):733–745. Alvarez, M. A., Rosasco, L., Lawre...

2020

[2] [2]

Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise.arXiv preprint arXiv:1706.03825. Štrumbelj, E. and Kononenko, I. (2014). Explaining prediction models and individual predictions with feature contributions.Knowledge and information systems, 41:647–665. Sundararajan, M., Taly, A., and Yan, Q...

arXiv 2017

[3] [10]

Sutton, R

MIT press Cambridge. Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation.Advances in neural information processing systems,

1999

[4] [11]

Verma, A., Murali, V., Singh, R., Kohli, P., and Chaudhuri, S. (2018). Programmatically interpretable reinforcement learning. InInternational conference on machine learning, pages 5045–5054. PMLR. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learn- ing.Machine learning, 8(3):229–256. Xu, T., Wang...

arXiv 2018

[5] [12]

and Stafford, R

Lever, G. and Stafford, R. (2015). Modelling policies in mdps in reproducing kernel hilbert space. InArtificial intelligence and statistics, pages 590–598. PMLR. Li, J., Kuang, K., Wang, B., Liu, F., Chen, L., Wu, F., and Xiao, J. (2021). Shapley counterfactual credits for multi-agent reinforcement learning. InProceedings of the 27th ACM SIGKDD Conference...

2015

[6] [13]

Across nonzero noise levels, RSA2C- CME shows markedly lower variability, with standard deviations from 24.75 to 43.06, whereas RSA2C-KME ranges from 46.74 to 56.03. For the mean return, the two variants are similar at light noise 0 and 0.001, and CME outperforms KME at higher noise 0.005 and 0.01, yielding 264.59 versus 242.89 and 254.85 versus 242.05, r...

[7] [15]

And we can estimate∇hJ(h,Σ)by sampling nstate-action pairs{s ι,a ι, rι,s ι+1}and approximating with the average

Note that the steepest ascent direction is a function inH. And we can estimate∇hJ(h,Σ)by sampling nstate-action pairs{s ι,a ι, rι,s ι+1}and approximating with the average. B.3 Compatible function approximation In this section, we demonstrate that the Advantage Critic and Value Critic satisfied the compatible function architecture. To begin with, we have b...

1999

[8] [16]

•L V: Lipschitz drift of the Value Critic w.r.t

σ with0< ν < σ≤1. •L V: Lipschitz drift of the Value Critic w.r.t. actor parameters:∥w⋆ V(h′)−w ⋆ V(h)∥ ≤LV∥h′ −h∥. Whenonlyscalarkernelsareinvolved, onemaysafelyreplaceM K, MΓ byM k. Foroperator-valuedkernels / vector-valued RKHS, keepMK (kernel boundedness) andMΓ (second-order operator bound) separate. C.2 Supporting Lemmas Lemma 1(Theorem 5 in Zhang et...

2020