Counterfactual Shapley Credit Assignment

Elias Bareinboim; Kaizhan-Lee; Mingxuan Li

REVIEW 3 major objections 5 minor 35 references

Counterfactual Shapley Credit Assignment

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that rewards can be redistributed as counterfactual Shapley values, giving dense, causal credit to actions while provably preserving the optimal policy.

desk verdict The empirical recipe is interesting and the writing is clear, but the paper's central theoretical claim—that the phi-MDP preserves optimal policies—is false under the self-baseline used in the algorithm, and fixing that will require reworking the theory sections substantially. read the letter →

arxiv 2607.16999 v1 pith:4GCHNYWX submitted 2026-07-18 cs.LG cs.AI

Mingxuan Li , Kaizhan-Lee , Elias Bareinboim This is my paper · ORCID

classification cs.LGcs.AI

keywords creditassignmentcounterfactualShapleyvaluereinforcementlearningcausalinferencepolicygradientrewardredistributiondelayedstructuralmodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the credit assignment problem in reinforcement learning can be solved by redistributing the total reward as per-action Counterfactual Shapley values (ϕ-values), which measure each action's causal contribution to the observed outcome through counterfactual simulation. The central claim is that this redistribution—encoded in a new MDP called the ϕ-MDP—gives dense learning signals that concentrate credit on genuinely causal actions, zero out luck, and preserve the original optimal policy. The paper proves gradient and optimal-policy equivalence, derives a consistent estimator with linear-time amortized cost, and integrates the values into a practical policy-gradient algorithm. If correct, this would replace the temporal-proximity assumption underlying most credit assignment methods with a causal standard, improving learning in sparse, stochastic, and delayed-reward settings.

What carries the argument

The central object is the Counterfactual Shapley Value (ϕ-value): for a trajectory, the Shapley value of each action with respect to the game f(z) = Y − Y_z, where Y is the observed discounted return and Y_z is the return of a counterfactual trajectory in which the actions in coalition z are replaced by a baseline policy. The paper computes these values through counterfactual simulation in an SCM representation of the MDP, reusing observed transitions when the counterfactual state-action pair matches and resampling otherwise. The ϕ-MDP, obtained by replacing the reward r(s,x) with r^ϕ_π(s,x) = E[ϕ_t | s_t=s, x_t=x], is the mechanism that turns these values into learning signals. The consiste

What would settle it

Take a simple two-step MDP where a shared noise variable influences both the transition and the reward, and where the state-action pair diverges in the counterfactual. Compute the exact counterfactual natural total effect by conditioning on the posterior noise P(U|τ) and compare it with the value produced by the paper's Algorithm 2, which resamples noise whenever the counterfactual pair differs from the observed pair. Any gap between the two for a fixed trajectory disproves the consistency claim; an experiment that measures this gap on a grid of noise strengths would settle it.

Watch

Extended reading notes

Core claim

The paper claims that the Counterfactual Shapley Value (ϕ-value), which is the Shapley value of a game whose value is the counterfactual natural total effect of a coalition of actions on the discounted return, constitutes a complete causal credit assignment function. It satisfies four desiderata: non-causes get zero credit, causes get nonzero credit, credit scales with departure from a baseline policy, and larger effects get larger credit. Building on this, the paper defines the ϕ-MDP, in which the environment reward is replaced by the per-step expected ϕ-value, and proves in Theorem 3.7 that this MDP has the same optimal policies as the original. The paper further claims that ϕ-PPO, the alg

Load-bearing premise

The load-bearing premise is that counterfactual trajectories can be simulated while holding the realized environment noise fixed at the values that produced the observed trajectory; if the noise cannot be reused or conditioned on, the computed ϕ-values do not match the defined counterfactual effects.

Editorial extensions

If this is right

If the optimal-policy equivalence holds, any policy-gradient method can be trained on ϕ-rewards instead of environment rewards without changing what the optimal policy is, while receiving dense per-step signal.
In environments where few actions are causal, the gradient variance is reduced by roughly the ratio of causal actions to horizon length, since non-causal actions receive zero credit.
In high-stochasticity environments, the shared exogenous noise cancels in the counterfactual difference, so the variance of each credit estimate scales with (1−ρ_Y), potentially much smaller than the return variance.
Delayed-reward credit reaches all timesteps in a single episode through counterfactual simulation, bypassing the O(T)-episode backward propagation of tabular TD(0).
The experimental claim is that ϕ-PPO converges where prior credit assignment methods fail, on the SkillLuck and Combinatorial Lock benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reliance on a resettable simulator means the framework is most directly applicable in model-based or simulated settings; a natural extension would be to pair it with a learned world model, but the paper does not test this.
Because ϕ-values are defined relative to a baseline policy, the credit attributions are not absolute properties of the environment; changing the baseline changes the causal story, which is a feature for explanation but a caution for comparing attributions across settings.
The variance-reduction argument assumes that counterfactual trajectories can be simulated under the same exogenous noise as the observed trajectory; if noise cannot be shared or correlated across steps, the empirical gains would shrink.
A testable extension would be to apply ϕ-values as an interpretability tool: the computed per-action scores could be used to audit whether a trained policy relies on skill or luck, without retraining.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing you need to know: Theorem 3.7 does not hold as stated. Under the self-baseline they use throughout training (pi_base = pi_theta), Shapley efficiency gives sum_t phi_t = Y - Y^1, where Y^1 is a fresh rollout of the same policy from the same initial state. So E_{pi_theta}[sum phi_t] = E[Y] - E[Y^1] = 0 for every theta. The phi-MDP objective is identically zero, every policy is a maximizer, and the claimed arg max equality with the original MDP fails whenever J(theta) is non-constant. The proof conflates the score-function identity E[Y^1 grad log pi] = 0 with grad E[Y^1] = 0; the latter is false when Y^1 depends on theta through pi_base. A one-step Bernoulli MDP makes the collapse explicit: E[Y^1] = theta, gradient 1, while the score identity still gives zero. This is not a minor gap—it removes the paper's headline guarantee. The gradient equivalence lemma (Lemma 3.5) may still be true as an unbiasedness statement for policy gradients, so the algorithm might be salvageable, but the phi-MDP as an MDP is degenerate. What is genuinely new and good: applying counterfactual Shapley values to temporal credit assignment is a natural and underexplored move; the lambda-bootstrapped estimator, the optimal coalition proposal in Theorem 4.2, and the PTR scheme are real additions. The SkillLuck and Combinatorial Lock experiments are well-designed and show clear advantages over a broad set of baselines. The paper is well-written and the limitations section is candid about simulator access and the gap between episode counts and total simulation budget. The other soft spots are real but secondary. Algorithm 2 resamples transition/reward noise when a counterfactual trajectory goes off-track, while Eq. (4) defines the NTE through the posterior P(U|tau) with fixed exogenous noise; the implemented estimator is not computing the defined quantity unless the environment is special. Theorem 3.2's D3 and D4 proofs are hand-wavy. No code or data are released, and the sample-complexity claims in Theorem 4.2 and Prop. 3.11 are far from the M=1 or M=16 used in practice. Bottom line: worth engaging with, but as a serious referee you should require a rewritten theory section, either a corrected Theorem 3.7 or an honest re-scoping to unbiased gradient estimation, plus a fix or rename for the counterfactual simulation, before any acceptance.

Referee Report

3 major / 5 minor

Summary. The paper proposes Counterfactual Shapley Credit Assignment, a reward-redistribution method for RL that attributes per-action credit via Shapley values of a counterfactual natural total effect (NTE) game. The authors define a φ-MDP, claim it preserves the original optimal policy (Theorem 3.7), derive an O(T)-per-episode estimator with an optimal coalition-sampling proposal, and introduce φ-PPO with Prioritized Trajectory Replay (PTR). Experiments on SkillLuck, Combinatorial Lock, and DoorKey benchmarks show improved sample efficiency over existing credit-assignment baselines.

Significance. If the core theoretical claims held, the paper would offer a principled causal solution to temporal credit assignment: dense, unbiased reward redistribution that does not alter the optimal policy. The paper also contributes a tractable estimator (amortized O(T) per episode), a variance-reduction proposal distribution, and a substantial empirical comparison. However, the central optimal-policy-preservation theorem is internally inconsistent with the paper's chosen self-baseline, and the proof conflates a score-function identity with the gradient of a θ-dependent reward. The empirical results are interesting and may be salvageable, but the theoretical framing as stated is not tenable.

major comments (3)

[Theorem 3.7 / Appendix D.3] The central optimal-policy-equivalence claim is false under the paper's self-baseline. Section 3.1 states 'We use self-baseline throughout training' (π_base = π_θ). By Shapley efficiency, Σ_t φ_t = Y − Y^{σ1}. Under self-baseline, Y^{σ1} is a fresh rollout of the same policy from the same initial state, so E_{π_θ}[Y^{σ1}] = E_{π_θ}[Y], hence E_{π_θ}[Σ_t φ_t] = 0 for every θ. The φ-MDP objective is identically zero, so every policy is optimal for it; the claimed equality of arg max sets in Theorem 3.7 fails whenever J(θ) is non-constant. The proof in Appendix D.3 argues that E[Y^1] has zero gradient because 'Y^1 is independent of the realized actions X_t, so E[Y^1∇logπ]=0'. This conflates the score-function identity for a fixed reward with the gradient of E[Y^1] when Y^1 itself depends on θ through π_base = π_θ. In a one-step Bernoulli MDP, E[Y^1]=θ, ∇E[Y^1]=1, while E[Y^1∇logπ]=0. Thus t
[Algorithm 2 / Eq. (4)] The counterfactual simulation in Algo. 2 resamples transition and reward noise whenever the counterfactual state-action pair differs from the observed pair (lines 10–14). Equation (4) defines the NTE using the posterior P(U|τ) with the realized exogenous u fixed. These are different operations unless the environment is deterministic or Assumption 3.3 is interpreted as an independent-noise-per-parent SCM representation. The text asserts that 'when parents differ, we resample' as a consequence of Assumption 3.3, but it never proves that Algo. 2 is a valid Monte Carlo estimator of Eq. (4). Since the consistency and bias results (Theorems 4.2 and 4.4) rest on this equivalence, the paper should provide a rigorous derivation or clarify the exact relationship between the simulated quantity and the defined NTE.
[Theorem 3.2 / Appendix C-D] Contribution 1 claims that φ-values satisfy the four causal credit-assignment desiderata D1–D4. The proof in Appendix D.3 is not self-contained: for D3 (Causal Normality) and D4 (Causal Effect Scaling) it invokes monotonicity properties of the NTE game without formal justification, and the core attribution framework is delegated to Lee et al. (2025), a same-group technical report. As a result, the paper does not independently establish that φ-values 'align precisely to the ground truth causes of task rewards'—a central interpretive claim. The authors should either provide a complete, self-contained proof or clearly mark Theorem 3.2 as an imported result and adjust the contribution claims accordingly.

minor comments (5)

[Theorem 4.2] The statement says 'variance-minimizing proposal', but the proof minimizes an upper bound on the second moment (uniform bound on Var[f(z)]) rather than the exact variance, which may depend on the coalition size k. The wording should be relaxed to 'minimizes a worst-case bound' or the proof should handle heteroskedastic f(z).
[Section 3.1 / Eq. (4)] The text says 'we condition on the observed trajectory, giving the conditional causal contribution f(z)=Y−Y^{σz}', but Eq. (4) defines an expectation over posterior and prior noise. These are not identical unless the expectation is implicit. Clarify whether f(z) is a random variable or an expectation.
[Notation] The symbol λ is used both for NTE mixing (Eq. 10) and GAE mixing (Eq. 13). Although Table 2 defines both, the dual use is confusing in Sections 4 and Appendix B. Consider renaming one of them.
[Appendix F] Assumption 3.3 is described both as a genuine assumption ('excluding correlated exogenous noise') and as a representation choice that 'does not constrain the MDP'. These two statements are in tension and should be reconciled.
[Figure 5] At σ=0 the paper admits several baselines match or exceed φ-PPO; the abstract's blanket claim of 'superior sample efficiency' is too strong. Consider qualifying the abstract to reflect the stochastic regime.

Circularity Check

2 steps flagged · score 6.0 of 10

Optimal-policy preservation reduces to the definition of φ under self-baseline; causal validity is imported from the authors' own technical report.

self citation load bearing [Section 3, Theorem 3.2; Contribution 1; Table 5 caption]
"We apply the Counterfactual Shapley Value (ϕ-value) framework Lee et al. (2025) to RL temporal credit assignment, showing that ϕ-values yield a reward redistribution that preserves the optimal policy and concentrates credit on causal actions."

The central claim that ϕ-values are causal (Theorem 3.2) is delegated to Lee et al. (2025), a Columbia technical report co-authored by two of the present authors (Kai-Zhan Lee and Elias Bareinboim). The desiderata D1–D4 are introduced as 'following the axiomatic framework of Lee et al. (2025)' (Table 5 caption), and no independent machine-checked or externally verifiable source is provided. The paper explicitly frames its contribution as applying that framework, so the load-bearing premise 'ϕ-values are causal' rests on a same-group self-citation rather than a first-principles derivation in this paper.
self definitional [Section 3.2 (undiscounted ϕ-return); Appendix D.3 (proof of Theorem 3.7)]
"Explicitly, by Shapley efficiency, Jϕ(θ)=J(θ)−Eπθ[Y^1]. The term Eπθ[Y^1] (expected baseline return under self-baseline) has zero gradient by the same argument as Lemma 3.5: Y^1 is independent of the realized actions X_t, so E[Y^1∇logπ]=0."

Under the self-baseline actually used throughout training (π_base=π), Y^1 is a fresh rollout of the same policy from the same initial state, so Eπθ[Y^1]=Eπθ[Y]. Combined with Shapley efficiency, Σ_t ϕ_t = Y−Y^1, this makes Eπθ[Σ_t ϕ_t]=0 for every θ. The φ-MDP objective is then identically zero, so the claimed arg-max equality is vacuous. The proof's step from E[Y^1∇logπ]=0 to ∇Eπθ[Y^1]=0 is invalid when Y^1 depends on θ through π_base=π. The 'optimal policy preservation' result is therefore an artifact of the definition of ϕ as Y−Y^1 plus the chosen baseline, not an independent theorem.

full rationale

The empirical portion of the paper (SkillLuck, Combinatorial Lock, Fork MDP, ablations) is a genuine benchmark comparison and is not circular; the estimator derivations (Theorems 4.2 and 4.4) are self-contained given the stated assumptions. However, the two central theoretical guarantees have circular or definitional structure. First, the causal validity of ϕ-values (Theorem 3.2) is imported from Lee et al. (2025), a same-group technical report, and the desiderata themselves are 'following the axiomatic framework of Lee et al. (2025)'. Second, the optimal-policy-preservation theorem is built into the construction: Shapley efficiency gives Σ_t ϕ_t = Y − Y^1, and under the self-baseline actually used, Eπθ[Y^1] = Eπθ[Y], forcing Eπθ[Σ_t ϕ_t] = 0 for all θ. The appendix proof's move from E[Y^1∇logπ]=0 to ∇Eπθ[Y^1]=0 is invalid when the baseline return is generated by the current policy. The limitations section flags only Assumption 3.3 and simulator access, not this collapse of the optimality claim. Because the main theoretical guarantee reduces to a definitional identity and the causal-validity premise is self-cited, while the empirical comparisons and estimation analysis retain independent content, the circularity is partial but material.

Assumptions & free parameters 5 free parameters · 6 assumptions · 2 invented entities

The ledger shows that the central results lean on the authors' own phi-value framework and on a strong, partially unflagged modeling choice about counterfactual resampling. The free parameters (M, lambda, baseline policy, PTR priorities) affect the empirical and consistency claims; the axioms include the MDP-as-SCM representation, simulator access, and the NTE/Shapley definition of 'credit'. No new physical entity is introduced; the invented objects are algorithmic.

free parameters (5)

Coalition samples M = M=1 (SkillLuck, Fork), M=16 (Combinatorial Lock)
The estimator's (epsilon,delta) guarantee (Thm 4.2) requires M=O(sigma^2/((1-gamma)^2 delta epsilon^2)); M is a user-set hyperparameter and experiments use small M, so consistency is asymptotic/idealized.
NTE mixing lambda = lambda=1.0 (pure MC)
Controls the bias-variance tradeoff of the bootstrap; Theorem 4.4 bias bound depends on lambda and is chosen by hand.
Self-baseline pi_base=pi = pi_base = current policy
Attribution is relative to the baseline policy; DoorKey results change substantially with baseline choice, so the causal-credit claim is baseline-relative.
GAE mixing lambda_phi = 0.95
Hyperparameter for phi-advantages; affects empirical convergence but not the central identities.
PTR priority exponent alpha_ptr = 0.6
The ablation claims PTR is necessary; the priority sharpness affects which trajectories are replayed, so this hand-set value matters for the empirical claims.

assumptions (6)

domain assumption Any MDP admits an SCM with independent exogenous noise and the same optimal policy (Assumption 3.3).
Used to justify re-sampling noise in Algo. 2; this is a representational assumption that is not derived from the MDP alone without loss.
ad hoc to paper The NTE game f(Z)=Y-Y^z plus Shapley axioms defines correct causal credit (D1-D4).
The desiderata and phi-value framework are imported from Lee et al. (2025); whether 'credit' should satisfy exactly these axioms is stipulated.
domain assumption Finite second moment E[Y_t^2] <= sigma^2 (Assumption 3.8).
Needed for variance bounds and the sample-complexity expressions.
domain assumption Simulator access for counterfactual rollouts.
Stated in Sec. F as a limitation; without a resettable simulator L3EST cannot be executed.
standard math Shapley value axioms, Markov property, score function identity, TD(lambda) convergence, Chebyshev and Cauchy-Schwarz inequalities.
Background results used throughout the proofs.
domain assumption PTR importance-sampling corrections anneal sufficiently fast to yield unbiased gradients.
PTR unbiasedness relies on IS weights correcting for non-uniform sampling; the practical annealing schedule is heuristic.

invented entities (2)

phi-MDP M^phi_pi
purpose: Reward-redistributed MDP used as a drop-in replacement for the original MDP in policy optimization.
Mathematical construction; its key property is exactly what the paper must prove, so it provides no independent evidence.
Prioritized Trajectory Replay (PTR)
purpose: Replay buffer prioritizing subtrajectories by |phi|; claimed necessary for the empirical results.
Algorithmic component with no external validation; the in-paper ablation is the only evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Counterfactual Shapley Credit Assignment." pith.science (2026). https://pith.science/paper/4GCHNYWX

@misc{pith2026260716999,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Shapley Credit Assignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GCHNYWX}},
  note         = {Machine review of arXiv:2607.16999}
}

abstract

The Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents. Existing frameworks, whether relying on temporal contiguity or hindsight-conditioned reward reweighting, frequently fail to attribute properly between an agent's policy (skill) and environmental stochasticity (luck). A principled approach to CAP must isolate the true causal drivers of observed outcomes from spurious correlations and environmental randomness. We introduce Counterfactual Shapley Credit Assignment, a novel framework grounded in causal theory that attributes credit and blame via the Counterfactual Shapley Value ($\phi$-value). By redistributing environmental rewards, $\phi$-values enhance temporal credit assignment across three critical dimensions: sparse causality, high stochasticity, and delayed rewards, all while preserving the optimal policy. We derive a consistent estimator that computes $\phi$-values efficiently, enabling a new class of policy gradient methods, $\phi$-PPO, combined with Prioritized Trajectory Replay (PTR). Empirical results demonstrate that $\phi$-values align precisely to the ground truth causes of task rewards with superior sample efficiency in challenging environments where prior state-of-the-art methods fail to converge.

Figures

Figures reproduced from arXiv: 2607.16999 by the authors.

**Figure 1.** Among all tested methods, only causal credit assignment correctly distinguishes Skill from Luck. Left: A SkillLuck MDP under a uniform random policy. The first action selects the branch; on the top branch, xN =1 yields reward 1 with certainty (“Skill,” red) while xN =0 yields 0 (gray). On the bottom branch, the reward is drawn from Bern(0.5) regardless of xN (“Luck,” blue). Center: Only our causal credit assignment … view at source ↗

**Figure 2.** Our counterfactual Shapley values assigns causal credits to each action via reward re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Counterfactual simulation (T = 5). Left: z = 0, all values reused, Y z = Y . Right: z = (0, 1, 0, 1, 1); intervention at t = 2 causes divergence, returns on-track at t = 4 (s z 4 = s4 by chance), diverges again at t= 5. D1 Causal Admissibility : Non-cause actions should be assigned zero credit; D2 Causal Power : Actions causing the rewards should be assigned non-zero credit; D3 Causal Normality : Actions against mor… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Three dimensions where ϕ-redistribution reduces variance. (a) Sparse causality. TD updates all T actions; ϕ updates only the k ′ causal actions, reducing variance by factor k ′/T. (b) High stochasticity. TD compares returns under different noise realizations (high vari…

**Figure 5.** Figure 5: Skill vs. Luck results (T=100, γ=0.999, M=1, n=50 seeds per condition). Left: MDP structure; the first action selects skill or luck, and only the final action on the skill branch affects the reward. (a) Success rate with Wilson 95% CIs across noise levels σ ∈ {0, 3, 10…

**Figure 6.** Figure 6: Combinatorial Lock results (T ∈ {5, 10, 15, 20, 25}, n=20 seeds, each seeded with one optimal trajectory). Left: MDP structure; CORRECT advances one step (zero reward) until the final transition yields Y =1 (discounted return γ T −1 , twice any shortcut’s γ T /2), whil…

**Figure 7.** Figure 7: DoorKey attributions (M=4096, γ=0.99, 20 seeds). Left: the agent (red) must collect the key, unlock the door, and reach the goal (green). Top right: uniform random baseline. Bottom right: navigation-only baseline that follows the shortest path ignoring walls. equally. …

**Figure 8.** Figure 8: Fork MDP results (n=1,000 runs per condition; shaded bands are 95% CIs). Left: MDP structure; only the first c actions affect the terminal reward. Top row (a–c): Success rate across noise σ, horizon T, and causal steps c. Bottom row (d–f): Mean episodes to convergence.…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 2 canonical work pages

[1]

Stratified sampling reduces variance in the fresh/replay ratio compared to per-trajectory Bernoulli sampling

and collect nfresh = clamp(⌊ξB⌋,1, B−1)fresh rollouts. Stratified sampling reduces variance in the fresh/replay ratio compared to per-trajectory Bernoulli sampling. Importance-sampling corrections.The IS correction depends on the trajectory source. Both fresh and replay trajectories use the PPO ratio for multi-epoch updates: rj,t =π θ(xj,t |s j,t)/πθold (...

2026
[2]

The exponent αptr controls priority sharpness; ϵptr prevents starvation

This prioritizes states where (i) actions have consistent causal impact (high |µ|), and (ii) the action choice matters but the policy has not yet converged (high σ with µ≈0 ). The exponent αptr controls priority sharpness; ϵptr prevents starvation. Priority decay.Priorities become stale as the policy improves. Each iteration, all priorities decay by pj ←γ...

2017
[3]

The action’s causal effect is ϕ= 1 with zero variance, while Var[Y] = 2500

Discount factorγ0.99 Trajectory length (chunk size)T chunk 128 Batch size (trajectories)B64 Policy epochsK4 Coalition samplesM1 NTE mixingλ1.0 (pure MC) GAE mixingλ ϕ 0.95 PPO clipϵ clip 0.2 Value loss coefficientc 1 0.5 ϕ-value loss coefficientc 2 0.5 Entropy coefficientc 3 0.01 Gradient clip normg max 0.5 Exploration decayγ ξ 0.99 PTR (Appendix D.5) Pri...

1992
[6]

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch

URL https://proceedings.mlr.press/v139/chang21b.html. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaugh...

2021
[7]

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal

URL https://proceedings.neurips.cc/paper/ 2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html. Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium,

2021
[10]

In contrast, ϕ-redistribution computes ϕt for all t from a single trajectory via O(T·M δ,ϵ) counterfac- tual simulations

φ-PPO (ours) REINFORCE PPO Figure 8:Fork MDP results( n=1,000 runs per condition; shaded bands are 95% CIs).Left:MDP structure; only the first c actions affect the terminal reward.Top row (a–c):Success rate across noise σ, horizonT, and causal stepsc.Bottom row (d–f):Mean episodes to convergence. In contrast, ϕ-redistribution computes ϕt for all t from a ...

2026
[12]

Hsiao-Ru Pan and Bernhard Schölkopf

URL http://arxiv.org/abs/ 1910.07113. Hsiao-Ru Pan and Bernhard Schölkopf. Skill or luck? return decomposition via advantage functions. InThe Twelfth International Conference on Learning Representations,

arXiv 1910
[14]

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen

URL https://arxiv.org/abs/2102.12425. Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling.Foundations and Trends in Machine Learning, 11(1):1–96,

arXiv

Show all 35 references

[16]

URL http://arxiv.org/abs/1511.05952. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by...

arXiv
[18]

Jordan, and Pieter Abbeel

John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. In Yoshua Ben- gio and Yann LeCun (eds.),4th International Conference on Learning Representations, ICLR 2016, San Ju...

2016
[19]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

URL http://arxiv.org/abs/1506.02438. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv
[20]

Lloyd S Shapley

URL http://arxiv.org/abs/ 1912.10944. Lloyd S Shapley. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker (eds.), Contributions to the Theory of Games II, pp. 307–317. Princeton University Press, Princeton,

1912 arXiv
[25]

Chao Yu, Jiming Liu, and Shamim Nemati

URLhttps://arxiv.org/abs/2505.09388. Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey,

arXiv
[27]

1, which controls bootstrapping depth)

to theϕ-MDP: Aϕ t = T−tX ℓ=0 (λϕ)ℓδϕ t+ℓ,(13) where λϕ ∈[0,1] controls the bias-variance tradeoff (distinct from λ in Algo. 1, which controls bootstrapping depth). Theϕ-return is Gϕ t = T−tX ℓ=0 ˆϕt+ℓ,(14) which serves as the training target for V ϕ ω (Eq. 22). Shapley efficie...

2018
[30]

Alice arriving

correct for the prioritized sampling distribution, ensuring unbiased gradient estimates. Hyperparameters are in Table 3; proofs in Appendix D.4. C Theory Details Definition C.1(Functional Dependence).Given world(M,u), the functional dependence ofYon Xunder baselinez= (x ′,z ′)...

2026
[31]

Reinforcement Learning Journal 2026 Table 2: Notation glossary Symbol Meaning SCM and MDP V,UEndogenous, exogenous variables τTrajectory:(s 1:T , x1:T , y1:T ) St, Xt, Yt State, action, reward at timet π(x|s)Policy (action distribution given state) γ∈(0,1)Discount factor THori...

2026
[32]

These samples are drawn independently of the observed actions X1:T

Under any baseline policy πbase (including self-baseline), the counterfactual Y 1 replaces all observed actions with fresh samples X 1 1:T ∼π base(· |S1 1:T ). These samples are drawn independently of the observed actions X1:T . We claim Y 1 ⊥X t |S t. By the Markov property o...

2026
[33]

Sampling selects (trajectory, starting timestep) pairs proportionally to pt using the Gumbel-max trick for efficient vectorized sampling

of squared Shapley estimates, αptr controls priority sharpness, and ϵptr prevents starvation. Sampling selects (trajectory, starting timestep) pairs proportionally to pt using the Gumbel-max trick for efficient vectorized sampling. When the buffer reaches capacity, the traject...

2026
[36]

Reinforcement Learning Journal 2026 Table 4: Credit assignment methods: expressions and descriptions

2For clarity, prior uses of the term ’counterfactual’ in this literature refer to hindsight conditioning not counterfactuals defined in the causal theory (Pearl, 2009). Reinforcement Learning Journal 2026 Table 4: Credit assignment methods: expressions and descriptions. Method...

2009
[1988]

DOI: 10.1023/A:1022633531479

ISSN 0885-6125. DOI: 10.1023/A:1022633531479. URL https: //doi.org/10.1023/A:1022633531479. Richard S Sutton and Andrew G Barto.Reinforcement Learning: An Introduction. A Bradford Book, second edition,

doi
[1992]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

ISSN 0025-1909. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Be...

1909
[1997]

DOI: 10.1109/9. 580874. Akash Velu, Skanda Vaidyanath, and Dilip Arumugam. Hindsight-DICE: Stable credit assignment for deep reinforcement learning,

doi
[1999]

URL https://doi.org/10.1016/ S0004-3702(99)00052-1

DOI: 10.1016/S0004-3702(99)00052-1. URL https://doi.org/10.1016/ S0004-3702(99)00052-1. Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: a scalable real-time architecture for learning knowledge from unsup...

doi
[2009]

Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni

DOI: 10.1017/CBO9780511803161. Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning.Transactions on Machine Learning Research,

doi
[2015]

DOI: 10.1038/nature14236. OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wo...

doi
[2016]

Hindsight credit assignment

Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, and Rémi Munos. Hindsight credit assignment. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’A...

2019
[2017]

DOI: 10.1038/nature24270

ISSN 1476-4687. DOI: 10.1038/nature24270. URLhttps://www.nature.com/articles/nature24270;. Richard S. Sutton. Learning to predict by the methods of temporal differences.Mach. Learn., 3(1):9–44, August

doi
[2018]

Michael Chang, Sid Kaushik, Sergey Levine, and Tom Griffiths

DOI: 10.1007/s11229-016-1247-1. Michael Chang, Sid Kaushik, Sergey Levine, and Tom Griffiths. Modularity in reinforcement learning via algorithmic independence in credit assignment. In Marina Meila and Tong Zhang (eds.),Proceedings of the 38th International Conference on Machi...

doi
[2019]

Counterfactual Shapley Credit Assignment Ilze Amanda Auzina, Joschka Strüber, Sergio Hernández-Gutiérrez, Shashwat Goel, Ameya Prabhu, and Matthias Bethge

URL https://arxiv.org/ abs/1806.07857. Counterfactual Shapley Credit Assignment Ilze Amanda Auzina, Joschka Strüber, Sergio Hernández-Gutiérrez, Shashwat Goel, Ameya Prabhu, and Matthias Bethge. Intrinsic credit assignment for long horizon interaction,

arXiv
[2020]

Reinforcement Learning Journal 2026 Supplementary Materials The following content was not necessarily subject to peer review

URLhttps://arxiv.org/abs/1908.08796. Reinforcement Learning Journal 2026 Supplementary Materials The following content was not necessarily subject to peer review. Appendix Contents A Counterfactual Simulation Bϕ-PPO C Theory Details D Proofs E Additional Experiments F Limitati...

1908 arXiv
[2021]

URL https://ojs.aaai.org/index.php/AAAI/article/view/16772

DOI: 10.1609/aaai.v35i7.16772. URL https://ojs.aaai.org/index.php/AAAI/article/view/16772. Sander Beckers and Joost Vennekens. A principled approach to defining actual causation.Synthese, 195(2):835–862,

doi
[2022]

URLhttps://dl.acm.org/doi/book/10.1145/3501714

ISBN 9781450395861. URLhttps://dl.acm.org/doi/book/10.1145/3501714. Sander Beckers. The counterfactual ness definition of causation.Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):6210–6217, May

doi
[2023]

Ratio measures how much Xt increased the proba- bility of observingU ′

P π(U ′ |S t, Xt) P π(U ′ |S t) −1U ′: encoding of a future step (S′, X′, Y′); typically U ′=Y ′ (reward). Ratio measures how much Xt increased the proba- bility of observingU ′. Estimated by supervised learning D. Ours: counterfactual simulation ϕ-valuesY−Y z,sharedU Replay t...

2025
[2025]

DOI: 10.1038/s41586-025-09422-z

ISSN 1476-4687. DOI: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z. Joseph Y Halpern.Actual causality. MiT Press,

doi
[2026]

Elias Bareinboim, Juan D Correa, Duligur Ibeling, and Thomas Icard

URL https://arxiv.org/abs/2202.06793. Elias Bareinboim, Juan D Correa, Duligur Ibeling, and Thomas Icard. On pearl’s hierarchy and the foundations of causal inference. InProbabilistic and causal inference: the works of judea pearl, pp. 507–556. Association for Computing Machin...

arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Stratified sampling reduces variance in the fresh/replay ratio compared to per-trajectory Bernoulli sampling

and collect nfresh = clamp(⌊ξB⌋,1, B−1)fresh rollouts. Stratified sampling reduces variance in the fresh/replay ratio compared to per-trajectory Bernoulli sampling. Importance-sampling corrections.The IS correction depends on the trajectory source. Both fresh and replay trajectories use the PPO ratio for multi-epoch updates: rj,t =π θ(xj,t |s j,t)/πθold (...

2026

[2] [2]

The exponent αptr controls priority sharpness; ϵptr prevents starvation

This prioritizes states where (i) actions have consistent causal impact (high |µ|), and (ii) the action choice matters but the policy has not yet converged (high σ with µ≈0 ). The exponent αptr controls priority sharpness; ϵptr prevents starvation. Priority decay.Priorities become stale as the policy improves. Each iteration, all priorities decay by pj ←γ...

2017

[3] [3]

The action’s causal effect is ϕ= 1 with zero variance, while Var[Y] = 2500

Discount factorγ0.99 Trajectory length (chunk size)T chunk 128 Batch size (trajectories)B64 Policy epochsK4 Coalition samplesM1 NTE mixingλ1.0 (pure MC) GAE mixingλ ϕ 0.95 PPO clipϵ clip 0.2 Value loss coefficientc 1 0.5 ϕ-value loss coefficientc 2 0.5 Entropy coefficientc 3 0.01 Gradient clip normg max 0.5 Exploration decayγ ξ 0.99 PTR (Appendix D.5) Pri...

1992

[4] [6]

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch

URL https://proceedings.mlr.press/v139/chang21b.html. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaugh...

2021

[5] [7]

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal

URL https://proceedings.neurips.cc/paper/ 2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html. Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium,

2021

[6] [10]

In contrast, ϕ-redistribution computes ϕt for all t from a single trajectory via O(T·M δ,ϵ) counterfac- tual simulations

φ-PPO (ours) REINFORCE PPO Figure 8:Fork MDP results( n=1,000 runs per condition; shaded bands are 95% CIs).Left:MDP structure; only the first c actions affect the terminal reward.Top row (a–c):Success rate across noise σ, horizonT, and causal stepsc.Bottom row (d–f):Mean episodes to convergence. In contrast, ϕ-redistribution computes ϕt for all t from a ...

2026

[7] [12]

Hsiao-Ru Pan and Bernhard Schölkopf

URL http://arxiv.org/abs/ 1910.07113. Hsiao-Ru Pan and Bernhard Schölkopf. Skill or luck? return decomposition via advantage functions. InThe Twelfth International Conference on Learning Representations,

arXiv 1910

[8] [14]

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen

URL https://arxiv.org/abs/2102.12425. Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling.Foundations and Trends in Machine Learning, 11(1):1–96,

arXiv