{"id":"f763057d-998e-4a9c-9418-2a726ae95afe","arxiv_id":"2506.05445","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A backdoor-adjusted SAC variant that substitutes pseudo-past variables inferred from the current state for the true past marginal, with empirical gains but an unjustified causal estimator.","lead":"DoSAC is a proposed extension of Soft Actor-Critic that uses a backdoor reconstructor to guess past state-action pairs from the current state, aiming to learn an interventional policy π(a|do(s)). The paper reports strong empirical gains in continuous control, but the implementation does not actually compute the backdoor adjustment it claims.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4)–Algorithm 1 substitute p(z|s) for the marginal p(z), so DoSAC's sampled policy is the observational π(a|s), not π(a|do(s)); the causal claim does not follow.","rationale":"The reader's verdict is REJECT with high confidence, and the identified weakness is indeed the load-bearing one. The paper's entire contribution reduces to estimating π(a|do(s)) through backdoor adjustment: if the estimator is actually sampling from p(a|s), the causal mechanism is absent, the theory in Lemmas 1–2 does not apply to the implemented algorithm, and the empirical gains cannot be attributed to confounding correction. The reader pinpointed Eq. (4) and Algorithm 1 as replacing the required marginal p(a_{t-1},s_{t-1}) with the conditional p_phi(a_{t-1},s_{t-1}|s_t). I agree with this diagnosis, and the analysis can be sharpened: when the learned conditionals are exact, the composition in Algorithm 1 equals the ordinary conditional distribution p(a|s) by the law of total probability. Hence the backdoor adjustment cancels itself out. This is a formal, internal inconsistency in the method, not merely a disagreement with an external consensus or a missing empirical comparison. The experimental setup further weakens support for the causal claim, since the injected additive action noise is not a confounder of the state-action relationship in the sense of the paper's causal graph, and no oracle or identified interventional policy is used to verify the target. However, the theoretical flaw is sufficient on its own. The recommended verdict remains REJECT, matching the reader's verdict; no adjustment is needed.","tokens_in":14551,"tokens_out":6909,"duration_ms":79388,"concrete_test":"Construct a small linear-Gaussian SCM with known conditionals, e.g., U ~ N(0,1), A_{t-1}=U+ε1, S_t=A_{t-1}+ε2, A_t=αS_t+βA_{t-1}+U+ε3. Compute three quantities analytically: (i) the true interventional policy E_{A_{t-1}}[p(a_t|s_t,A_{t-1})] using the marginal over A_{t-1}; (ii) Algorithm 1's estimator E_{A_{t-1}|s_t}[p(a_t|s_t,A_{t-1})]; (iii) the observational policy p(a_t|s_t). If (ii) equals (iii) and differs from (i), the implemented estimator performs no backdoor adjustment. This test requires no training because the exact conditionals are known from the SCM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central flaw is in Section 4.3, Eq. (4). The paper writes p(a_{t-1},s_{t-1}) = ∫ p(a_{t-1},s_{t-1}|s_t)p(s_t)ds_t, which is a valid marginalization identity only if the expectation over s_t is taken. Algorithm 1 does not do this: for a query state s it draws (a_{t-1},s_{t-1}) ~ q_phi(·|s_t=s) and then samples a ~ p_theta(·|s,a_{t-1},s_{t-1}). The resulting action distribution is ∫ p_theta(a|s,z) q_phi(z|s) dz, whereas the backdoor formula in Eq. (3) requires ∫ p_theta(a|s,z) p(z) dz. If q_phi and p_theta are the true conditionals, the implemented estimator equals ∫ p(a,z|s) dz = p(a|s) by the law of total probability. Thus the method estimates the observational policy, not the interventional policy; conditioning on s_t reintroduces the very spurious dependence that backdoor adjustment is meant to remove. The same conditional sampling appears in the Q-target update in Eqs. (7)–(8), so the entire training objective optimizes the wrong quantity. Consequently, the claim that DoSAC removes hidden confounding is unsupported, and the reduction argument in Lemma 2, while accidentally true in the confounder-free case because the composition then equals p(a|s), does not establish that the method computes π(a|do(s)) in confounded settings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DoSAC, an extension of Soft Actor-Critic that aims to learn the interventional policy π(a|do(s)) rather than the observational policy π(a|s), in order to remove bias from hidden confounders in off-policy RL. The method introduces a learnable Backdoor Reconstructor q_φ(a_{t-1},s_{t-1}|s_t) that samples pseudo-past variables, uses these samples in a backdoor-adjustment formula, and adds a causal entropy regularizer to the SAC objective. The authors report experiments on continuous control benchmarks with synthetically injected confounding, comparing DoSAC against SAC, ACE, TD3, DDPG, PPO, and RPO, and claim both theoretical guarantees (Lemmas 1 and 2) and improved robustness and generalization.","tokens_in":14893,"tokens_out":10089,"duration_ms":116940,"significance":"If the central claim were correct, DoSAC would be a practically valuable contribution: a method that estimates interventional policies from observational replay data without access to true confounders or causal labels, integrated into a standard actor-critic framework. The paper addresses an important problem, and the experimental comparison covers several baselines and a sensitivity analysis over confounder strength. However, the load-bearing estimation step in Section 4.3 is not a valid backdoor adjustment: the implemented sampler replaces the required marginal p(a_{t-1},s_{t-1}) with a conditional q_φ(a_{t-1},s_{t-1}|s_t), and the causal entropy used in the Q-target is the entropy of the conditional actor rather than of the interventional policy. The paper therefore does not establish that DoSAC estimates π(a|do(s)), and the empirical results do not compensate for this because no experiment checks the estimated quantity against a known causal target. The theoretical and algorithmic core would need to be redesigned, not locally corrected.","major_comments":[{"comment":"The implemented estimator does not compute the backdoor-adjusted policy. Eq. (3) requires an expectation over the marginal p(a_{t-1},s_{t-1}). Eq. (4) is a valid marginalization only if s̃_t is drawn from p(s̃_t) and then (a_{t-1},s_{t-1}) is drawn from p(·|s̃_t). The paper instead says it approximates this by setting s̃_t = s_t and learns q_φ(a_{t-1},s_{t-1}|s_t), and Algorithm 1 draws (ã, s̃) ∼ q_φ(·|s) for the query state s only. Since s_t is a descendant of (a_{t-1},s_{t-1}) under the assumed dynamics, q_φ(z|s) is generally unequal to p(z), and the action distribution produced is ∫ p_θ(a|s,z) q_φ(z|s) dz, which equals the observational p(a|s) when the learned conditionals match the data, not π(a|do(s)). Conditioning on the current state thus reintroduces the dependence that backdoor adjustment is supposed to remove. Section 6 concedes that pseudo-past inference 'may prove insufficient,' but the issue here is not approximation error; it is that Eq. (4) is not implemented as stated. This is the load-bearing step of the paper, so the claim that DoSAC estimates interventional policies is unsupported.","section":"Section 4.3, Eq. (4) and Algorithm 1"},{"comment":"The quantity used as causal entropy in the Q-target is not the entropy of the interventional policy. In Eq. (8), the bonus is α log p(ã′|s′, s̃, ã), where (s̃, ã) ∼ q_φ(·|s′) and ã′ ∼ p_θ(·|s′, s̃, ã). This is the log-density of the conditional actor for one sampled z, not log π(ã′|do(s′)), whose density under Algorithm 1 is the mixture ∫ p_θ(a|s′,z) q_φ(z|s′) dz. Consequently the objective optimized by the algorithm is not Eq. (2), and H(a|do(s)) from Eq. (1) is never actually computed. This compounds the first issue throughout the policy evaluation and policy improvement steps.","section":"Section 4.3, Eqs. (7)–(9)"},{"comment":"No loss function or training objective is given for the Backdoor Reconstructor q_φ. The text states it is 'a neural network trained to approximate the inverse dynamics,' and Algorithm 1 relies on samples from q_φ, but the only explicit loss in the paper is the Q-loss in Eq. (9), which does not constrain q_φ to estimate p(z|s) or any marginal p(z). Without a defined objective for q_φ, the procedure is underspecified and cannot be reproduced or evaluated as written.","section":"Section 4.3, 'Training the model'"},{"comment":"The empirical evaluation does not test whether the learned policy equals or approximates the interventional policy. Confounding is injected as additive action noise and methods are compared by return, but there is no experiment with a known causal target—for example, a synthetic SCM where π(a|do(s)) can be computed exactly—or a check that the backdoor adjustment removes a specific spurious correlation. Because the theory in Section 4 is in question, the improved returns alone do not establish that any causal adjustment is taking place; they could result from the added entropy regularizer or extra model capacity.","section":"Section 5 and Appendix A"}],"minor_comments":[{"comment":"The caption says 'All agents were trained in the presence of hidden confounders, but evaluated in clean environments,' which appears to be copied from Table 1; Table 2 reports evaluation with confounders, so the caption and the surrounding sentence should be corrected.","section":"Table 2 caption"},{"comment":"There are typographical errors: 'Humanroid' should be 'Humanoid' in Section 5, and 'inerventional' should be 'interventional' in Eq. (2).","section":"Section 5 and Eq. (2)"},{"comment":"The notation is ambiguous: the dummy variable s̃_t is not distinguished from the query state s_t, and it is unclear how the outer expectation over p(s̃_t) is estimated from the replay buffer; this should be clarified independently of the correctness issue.","section":"Eq. (4)"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: DoSAC is a genuinely new mechanism—a Backdoor Reconstructor that infers pseudo-past variables and feeds them into SAC—but the central derivation doesn't hold up. The backdoor formula in Eq. (3) requires the marginal p(a_{t-1},s_{t-1}), and Eq. (4) then Algorithm 1 replace that marginal with the conditional p_phi(a_{t-1},s_{t-1}|s_t). Sampling (a',s') ~ p_phi(·|s) and then a ~ p_theta(a|s,a',s') gives the composition ∫ p(a|s,z)p(z|s)dz = p(a|s). So the method is estimating the observational policy, not the interventional one. The conditioning on s_t reopens the backdoor path rather than blocking it.\n\nThe paper does some things well: it positions itself clearly against DOVI and ACE, the end-to-end training is straightforward, and the reconstructor idea is worth thinking about. If you instead sampled the past from a distribution that approximates the true marginal (e.g., by first drawing a state from the replay buffer and then generating z), the backdoor adjustment would be meaningful. Lemma 2 is correct but says little about the confounded case; Lemma 1 is a standard soft-policy-improvement argument with a causal label.\n\nThe experiments are not persuasive on the causal point. Adding i.i.d. noise to the action after the policy selects it does not create a backdoor path between s_t and a_t—the noise is not a common cause of both. The positive returns may come from the extra stochasticity or the architecture, not from deconfounding. The paper's own limitation paragraph concedes that pseudo-past inference may be insufficient, which is exactly the weakness.\n\nMy take: I wouldn't cite this version, but I think it deserves referee time. The flaw is substantial but potentially fixable, and the problem is timely. A serious reviewer will catch the conditional/marginal slip, and the authors will need to either change the sampling scheme or reposition the contribution. It's a reasonable reading-group example of how causal inference gets misapplied in deep RL.\n\nBest,","headline":"DoSAC's backdoor adjustment is built on a conditional-marginal mix-up: Eq. (4) samples the past from p(z|s), so the method estimates the observational policy, not the interventional one.","tokens_in":15479,"tokens_out":5596,"would_cite":false,"duration_ms":63566,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DoSAC: RL policies that act on causal effects, not correlations.","keywords":["causal reinforcement learning","backdoor adjustment","interventional policy","hidden confounders","soft actor-critic","do-calculus","off-policy learning","causal entropy"],"falsifier":"In a synthetic MDP with a known latent confounder, compute the true $\\pi(a|\\mathrm{do}(s))$ by physically intervening on the state, then compare it with the distribution produced by DoSAC's two-stage sampler (Reconstructor followed by Interventional Actor). If the two distributions diverge systematically as the dependence of $s_t$ on $(a_{t-1}, s_{t-1})$ increases, the reconstructor is not implementing backdoor adjustment and the central claim is falsified.","tokens_in":14272,"feed_emoji":"🧠","tokens_out":6020,"duration_ms":61087,"temperature":0.7,"pith_summary":"This paper argues that hidden confounders that influence both states and actions bias policies learned by standard reinforcement learning, and that the bias can be removed by learning the interventional policy $\\pi(a|\\mathrm{do}(s))$ instead of the observational policy $\\pi(a|s)$. The proposed method, DoSAC, extends Soft Actor-Critic with a backdoor adjustment: a learnable Backdoor Reconstructor infers pseudo-past state-action pairs from the current state, and these are used as the conditioning covariates in the backdoor formula, so the policy can be sampled from observational replay-buffer data alone. The authors claim this yields policies that are robust and generalizable under confounded training, and that the method reduces exactly to standard SAC when no confounders exist.","feed_headline":"DoSAC: RL policies that act on causal effects, not correlations","feed_subtitle":"Soft Actor-Critic gains a Backdoor Reconstructor, targets π(a|do(s)), and returns to plain SAC when no confounding exists.","key_machinery":"The backdoor adjustment formula is the load-bearing identity: $\\pi(a|\\mathrm{do}(s)) = \\mathbb{E}_{p(a_{t-1}, s_{t-1})}[p(a|s, a_{t-1}, s_{t-1})]$, which turns an intervention into an average of observational conditionals. DoSAC's new component is the Backdoor Reconstructor, a neural network $p_\\phi(a_{t-1}, s_{t-1}|s_t)$ trained to produce pseudo-past variables from the current state, which the paper treats as a sampling proxy for the marginal $p(a_{t-1}, s_{t-1})$. Together with the Interventional Actor $p_\\theta(a_t|s_t, a_{t-1}, s_{t-1})$, the pair enables forward sampling from $\\pi(a|\\mathrm{do}(s))$ using replay-buffer data.","core_discovery":"The central claim is that the interventional policy $\\pi(a_t|\\mathrm{do}(s_t))$ can be estimated in an off-policy actor-critic setting without access to true confounders or causal labels. Using the backdoor criterion, the paper writes $\\pi(a_t|\\mathrm{do}(s_t)) = \\mathbb{E}_{p(a_{t-1}, s_{t-1})}[p(a_t|s_t, a_{t-1}, s_{t-1})]$, and replaces the marginal over the past with samples from a learned conditional $p_\\phi(a_{t-1}, s_{t-1}|s_t)$, called the Backdoor Reconstructor. Actions are then drawn by first sampling a pseudo-past pair and then sampling from the conditional actor. The paper claims that this two-stage sampling estimates the interventional policy, and that the resulting soft actor-critic objective with causal entropy generalizes SAC and recovers it exactly in clean environments.","pith_inferences":["Editorial extension: the identification step replaces a marginal with a conditional, so the unbiasedness result is an empirical hypothesis unless the proxy is shown to be faithful; the paper's own Lemma 2 only covers the clean case.","Editorial extension: the same Backdoor Reconstructor could be dropped into other off-policy actor-critic algorithms, since the adjustment is not specific to the SAC objective.","Editorial extension: the experiments inject additive Gaussian confounders; testing temporally correlated or structured confounders would show whether the method handles the non-stationary confounding the paper lists as a limitation."],"forward_implications":["Policies learned with DoSAC are invariant to the confounder that generated the training data, so they transfer to settings where the confounder is absent or different.","The method integrates into standard SAC pipelines with no additional data requirements, only two extra learned distributions.","When no confounders are present, DoSAC reduces exactly to SAC, so the causal machinery introduces no overhead in clean environments.","The causal entropy bonus $H(a|\\mathrm{do}(s))$ replaces the standard entropy bonus and drives exploration toward actions that are robust to confounding.","Empirically, DoSAC reports higher returns than SAC and the ACE baseline in confounded continuous-control benchmarks, both under persistent confounding and in clean evaluations."],"supporting_citations":[{"why":"Supplies the do-calculus and backdoor criterion that the method uses to convert the interventional policy into an average of observational conditionals.","marker":"Pearl 2009b"},{"why":"Provides the Soft Actor-Critic objective and entropy-regularized framework that DoSAC extends with causal entropy and backdoor adjustment.","marker":"Haarnoja et al. 2018"},{"why":"Theoretical backdoor-adjusted causal RL method (DOVI) that motivates the practical implementation DoSAC claims to provide.","marker":"Wang et al. 2021"},{"why":"The closest causality-aware baseline, ACE, which DoSAC compares against and goes beyond by handling hidden confounding.","marker":"Ji et al. 2024"}],"fun_headline_variants":["DoSAC backdoor-adjusts SAC to learn causal policies","Interventional policies for RL without confounder labels","Estimating do-interventions in Soft Actor-Critic","Causal effect estimation for reinforcement learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that sampling pseudo-past variables from the learned conditional distribution $p_\\phi(a_{t-1}, s_{t-1}|s_t)$ is an acceptable stand-in for the marginal $p(a_{t-1}, s_{t-1})$ required by the backdoor formula; because the current state is itself caused by the past, these two distributions generally differ, and if the difference is large the algorithm is not actually estimating the interventional policy.","fun_headline_variants_meta":{"raw":{"variants":["DoSAC backdoor-adjusts SAC to learn causal policies","Interventional policies for RL without confounder labels","Estimating do-interventions in Soft Actor-Critic","Causal effect estimation for reinforcement learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3698,"prompt_tokens":943,"completion_tokens":2755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2694}},"tokens_in":559,"tokens_out":2755,"duration_ms":24643,"temperature":1.0,"reasoning_tokens":2694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:28:17.410248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a synthetic MDP with a known latent confounder, compute the true $\\pi(a|\\mathrm{do}(s))$ by physically intervening on the state, then compare it with the distribution produced by DoSAC's two-stage sampler (Reconstructor followed by Interventional Actor). If the two distributions diverge systematically as the dependence of $s_t$ on $(a_{t-1}, s_{t-1})$ increases, the reconstructor is not implementing backdoor adjustment and the central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Soft Actor-Critic objective and entropy-regularized framework that DoSAC extends with causal entropy and backdoor adjustment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Theoretical backdoor-adjusted causal RL method (DOVI) that motivates the practical implementation DoSAC claims to provide."}],"review_version":1}