A Graphical Approach to State Variable Selection in Off-policy Learning

Joakim Blach Andersen; Qingyuan Zhao

REVIEW 2 major objections 4 minor 4 references

A Graphical Approach to State Variable Selection in Off-policy Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper gives graphical identification criteria for off-policy learning: a state set satisfying nested states, memorylessness, and a dynamic back-door condition suffices to identify the distribution under any adaptive policy from data…

desk verdict A genuinely useful unification of state selection in off-policy learning, but the proof chain has a load-bearing definitional gap in the m-separation calculus. read the letter →

arxiv 2501.00854 v1 pith:XW6L2UAW submitted 2025-01-01 stat.ME cs.LG

Joakim Blach Andersen , Qingyuan Zhao This is my paper · ORCID

classification stat.MEcs.LG MSC 62D2062M05

keywords off-policylearningstatevariableselectionacyclicdirectedmixedgraphsm-separationdynamictreatmentregimesMarkovdecisionprocessescausalidentification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy learning asks whether the value of a policy of interest can be estimated from data generated under a different policy. This paper claims a precise graphical answer: choose state variables that satisfy a nested-states condition, a memorylessness condition, and a dynamic back-door condition, and the joint distribution of states, actions, and rewards under any adaptive policy is identified by an explicit formula. The result matters because it unifies two literatures that treat the same problem differently: dynamic treatment regimes in statistics and offline reinforcement learning in computer science. It also gives practical guidance for state variable selection in real-world sequential decision problems such as container-logistics pricing. The simulation study shows that a wrong state choice can make a standard reinforcement-learning algorithm converge to a policy worse than the baseline policy.

What carries the argument

The central machinery is the acyclic directed mixed graph (ADMG) with m-separation as its independence criterion, together with the dynamic single-world intervention graph (d-SWIG) representation of policy interventions. m-separation asks whether every path between two variable sets is blocked by a conditioning set, and is the graphical version of conditional independence used in Assumptions 1-3. The d-SWIG $G(g)$ is obtained by splitting each decision vertex into a potential outcome $A_t(g)$, which inherits outgoing edges, and a natural counterfactual $A_t^-(g)$, which inherits incoming edges. Proposition 2 says the distribution of $(V(g), A^-(g))$ is global Markov with respect to $G(g)$; this is what converts the m-separations in dynamic unconfoundedness and memorylessness into the conditional independences of Lemmas 5 and 6. Those two lemmas carry the proof of the recursive equality in Theorem 7, which iterates to give Theorem 1.

What would settle it

Run a simulation from a small ADMG with known structural equations that satisfies Assumptions 1-3, record data under a null policy, predict the target-policy distribution from the formula, and compare against direct simulation of the target policy; a mismatch would refute the theorem. Alternatively, exhibit an ADMG satisfying the three assumptions whose d-SWIG distribution violates the global Markov property.

Watch

Extended reading notes

Core claim

The central claim is that the value of an adaptive policy can be identified from observational data whenever the state variables satisfy three graphical conditions: nested states (Assumption 1), memorylessness (Assumption 2), and the dynamic back-door criterion (Assumption 3), together with dynamic consistency and positivity. Theorem 1 states that under these conditions, and with intermediate rewards contained in states, the joint distribution of all states, actions, and rewards under the target policy $g$ equals $P(r_{T+1} \mid a_T, s_T) \prod_{t=1}^T g(a_t \mid s_t) P(n_t \mid a_{t-1}, s_{t-1})$, where $P$ is the observational distribution under a null policy. The proof works by recursion: Theorem 7 rewrites the conditional distribution of future variables under a sub-policy using dynamic consistency, unconfoundedness, and the Markov property, so that each step peels off the policy's action density and the observational transition density. This generalizes the static back-door criterion and subsumes the g-computation formula for dynamic treatment regimes.

Load-bearing premise

The load-bearing premise is that the d-SWIG distribution satisfies the global Markov property, a result imported from an unreviewed preprint; if that theorem has hidden conditions, the main identification proof collapses even if the formula is true.

Editorial extensions

If this is right

When the state set satisfies the three criteria, the value of any adaptive policy can be estimated consistently from data generated by a different policy without additional assumptions beyond dynamic consistency and positivity.
The framework recovers Robins' g-formula for dynamic treatment regimes as a special case, so the dynamic treatment regime identification logic is subsumed by the graph conditions.
For Markov decision processes, the paper makes explicit the randomized-decisions assumption that is usually implicit, and it explains why partially observed MDPs are generally not identified.
When previous actions are excluded from the state variables, verifying only the immediate back-door condition is enough, which simplifies state selection in common MDP-like settings.
The pricing simulation shows that using a state set that violates the criteria can lead a policy-iteration algorithm to learn a policy with negative regret relative to the null policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension, not developed in the paper, is to turn the three graphical conditions into a search procedure that selects a valid state set from candidate variables algorithmically.
Because the identification formula is nonparametric, it gives a template for sequential regression estimators of policy value; whether such estimators inherit gains from the Markov and back-door structure is left open.
The contrast between Assumption 3 and the weaker dynamic unconfoundedness suggests that identification may be policy-dependent: a graph that blocks identification for one policy can still identify another, and exploiting this selectively has not been explored.
The graph conditions also suggest empirical diagnostics: one could test the implied conditional independences on observed data before committing to a state set, and a failure would signal non-identifiability rather than just finite-sample error.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Here is my take on arXiv:2501.00854. The paper offers a graphical framework for selecting state variables in sequential off-policy learning, spanning DTRs and offline RL. The dynamic back-door criterion, the nested-states and memorylessness conditions, and the warning about latent projections creating "phantom confounding" are genuinely new and practically useful. The container-logistics simulation makes the stakes concrete: using a state that violates the criteria can produce policies worse than the null policy. Credit where due—this is a thoughtful unification, not a repackaging.

The soft spot is more than cosmetic. Section 2.2 defines m-separation using walk-blocking: a collider not in the conditioning set blocks a walk. Section 1 uses the standard ancestral definition: a collider blocks only if it is not an ancestor of the conditioning set. These are not equivalent. Example: A<->C<->B with C->D, conditioning on D. Standard m-separation says A and B are connected because C is an ancestor of D; the paper's walk definition declares them separated because C is not in D. The proofs of Lemmas 3 and 4, and hence Lemmas 5, 6, Theorem 7, and Theorem 1, are carried out with the walk notion. Since Assumptions 2 and 3 are stated in the ancestral sense, the implication from graphical separation to conditional independence is not established as written. This is a load-bearing flaw, not a typo. It might be repairable—rework the lemmas with the correct ancestral-blocking calculus—but the current proof chain does not go through.

A secondary weakness is Proposition 2, the global Markov property for d-SWIGs, which is imported from Zhao (2024b), an unreviewed preprint. The main theorem leans on it. The authors should either give a self-contained proof or include a full derivation in the supplement. Also, the simulation is not fully reproducible as shipped: the graph-simulator package is cited without a version and the full code is not included. Minor, but worth asking for.

Who is this for? Researchers working on identification in DTRs and anyone in RL who cares about when off-policy evaluation is legitimate. It deserves a serious referee, not a desk reject. I would send it out with a clear instruction to scrutinize the m-separation equivalence and the dependence on Zhao (2024b). If the authors close the definitional gap, this becomes a solid paper.

Referee Report

2 major / 4 minor

Summary. This paper develops a graphical framework for off-policy identification in sequential decision processes. With state sets S_t, decision variables A_t, and reward variables R_t, the authors propose three assumptions—nested states, memorylessness, and a dynamic back-door criterion—and argue that, under dynamic consistency and positivity, the joint distribution of states, actions, and rewards under an adaptive policy g is identified by a g-computation-type formula from the observational distribution. The paper then relates dynamic unconfoundedness to sequential ignorability in dynamic treatment regimes, discusses the implicit randomized-decision assumption in MDP/RL practice and the general non-identifiability of POMDPs, and presents a dynamic pricing simulation in which policy iteration with a state set violating the criteria produces worse policies than the null policy.

Significance. The main result, if correct, is a useful unification: it gives explicit, checkable graphical conditions that span both DTR and MDP settings and clarifies why POMDPs are generally not identified without additional structure. The recursive proof via d-SWIGs is natural, and the simulation study with a realistic container-pricing environment and detailed parameters is a practical asset. The paper also makes the important point that state choice is part of the identification problem, not merely a modeling convenience. The central limitation is that a load-bearing proof step is imported from an unreviewed preprint, which makes independent verification difficult.

major comments (2)

[Appendix B, proof of Proposition 2] The proof of Proposition 2 is a citation to Theorem 3 of Zhao (2024b), an unreviewed arXiv preprint. This proposition is the only bridge from m-separation in G(g) to the conditional independence statements in Lemmas 5 and 6, which in turn drive Theorem 7 and Theorem 1. The manuscript therefore has a load-bearing proof step that is not self-contained. I recommend either proving the needed global-Markov statement in the supplement or restating the cited theorem in full and verifying that its hypotheses apply to the d-SWIG with natural counterfactuals A^-(g) and policy vertices A(g).
[Section 1.1, Theorem 1] The displayed formula in Theorem 1 has a notational mismatch: the left-hand side conditions on A_T(g)=a_T, a single decision variable, while the right-hand side is a product over t=1,...,T involving all actions a_t. The left-hand side should be \bar A_T(g)=\bar a_T, the full action history. As printed, the central identification formula is not dimensionally consistent, and the subsequent remark claiming that the left side contains all decision variables is only true with the overline correction.

minor comments (4)

[Section 2.2] The walk-based blocking definition is easy to misread as conflicting with the ancestral-blocking definition in Section 1; the equivalence relies on allowing non-simple walks, which can pass through descendants of colliders. For example, with A<->C<->B and C->D, conditioning on L={D}, the walk A<->C->D<-C<->B is unblocked, so A and B are m-connected under both definitions. A sentence making this role of non-simple walks explicit would prevent confusion.
[Table C.1] The scenario labels in Table C.1 all appear as \rightarrow G with no distinction between G_1, G_2, and G_3, even though Section 6.2 defines different arrow types for the three scenarios. Please render the arrows consistently so that the parameter columns can be matched to the graphs.
[Table 1] The column 'Regret (%)' appears to report improvement relative to the null policy, with negative values indicating worse-than-null performance; please define this quantity explicitly, since standard regret is nonnegative.
[Throughout] There are numerous typographical errors that should be corrected in revision, including 'resemblence', 'identifiying', 'denscendants', 'sophiscated', 'dicussion', and 'oer' (Section 6.3).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is derived from explicit graphical separations via g-computation, not assumed as an input.

full rationale

The paper's identification argument is a standard g-computation/recursive factorization proof: Theorem 1 follows from Theorem 7, which follows from Lemmas 5 and 6, which translate the graphical m-separations in Assumptions 2 and 3* into conditional independences in d-SWIGs. The identification formula is not assumed by any assumption or fitted to data; the assumptions are graphical separation conditions, and the theorem derives an observational factorization for the policy intervention distribution. The main self-citations (Zhao 2024a,b; Guo and Zhao 2023) supply general graphical-model infrastructure such as the NPSEM-to-nested-Markov implication and m-separation equivalences, not the paper's target identification result. These are prior technical results with stated mathematical content, not a renaming of the paper's conclusion. The reviewer-level concern that the two m-separation definitions in Sections 1 and 2.2 may not be equivalent is a correctness or verification issue, not circularity; likewise, reliance on an unreviewed preprint for Proposition 2 is an evidence-quality concern. No fitted parameter is relabeled as a prediction, and no load-bearing step reduces by construction to its own input.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The theoretical identification result uses no fitted parameters; all free parameters listed are simulation-only and chosen by hand. The main proof depends on standard ADMG/NPSEM axioms plus an imported theorem from an unreviewed preprint (Zhao 2024b). No new physical or statistical entities are postulated.

free parameters (7)

Vessel capacity C = 6
Hand-chosen booking cap in the simulation; not part of the identification theorem.
Demand persistence p_D = 1 (G), 0.9/0.5/0.1 (G1)
Controls the strength of the memorylessness violation in the macro trend scenario.
Demand estimate accuracy p_Dhat = 0.25
Controls quality of the observable demand estimate; hand-chosen.
Booking noise probabilities p_B1, p_B2 = 0.15
Hand-chosen randomization for bookings, ensuring positivity.
Log-intensity coefficients beta^i_j = See Table C.1 (e.g., beta^1_1 = -0.65, beta^1_3 in {0,1,3,5})
Hand-chosen to create unmeasured confounding and retrospective price effects in scenarios G2 and G3.
Price update coefficients alpha^i_j, xi^i_j = See Table C.1 (alpha^2_4 = -1 in G2; xi^1_0 = -1, xi^1_1 = 1 in G3)
Hand-chosen to produce the price-update behaviors and competitor-price dependence.
Price randomization probabilities p_A1, p_A2, p_Ac,1 = 0.15, 0.15, and 1/0.5/0.25/0.1 across scenarios
Hand-chosen to ensure positivity and model the quality of competitor price intelligence.

assumptions (6)

domain assumption NPSEM with independent noise (Eq. 2): each vertex is a deterministic function of parents plus independent noise, and noise is global Markov w.r.t. the bidirected subgraph.
This is the structural causal model underlying the SWIG construction; it is standard in the ADMG literature but is a modeling assumption, not a theorem.
domain assumption Global Markov property of d-SWIGs (Proposition 2), imported from Theorem 3 of Zhao (2024b).
This black-box result translates m-separations in the intervened graph into conditional independences. It is load-bearing for the proof of Theorem 7.
domain assumption Dynamic consistency (Assumption 4): the recursion in Eq. (3) relating sub-policy interventions.
Assumed in the main theorem; Proposition 3 argues it follows from the NPSEM, but the theorem states it as an assumption.
domain assumption Positivity (Assumption 5): P(A_t=a_t|S_t=s_t) > 0 for all actions and states.
Standard overlap condition needed to divide by observed action probabilities in the g-computation factorization.
domain assumption Rewards contained in states (R_t subset of S_t for t <= T).
Used to factor intermediate reward innovations; the paper notes it can be dropped for marginal final-reward identification.
standard math Acyclicity of the causal graph and absence of directed cycles in the ADMG.
Standard definition of an ADMG; required for topological order and d-SWIG construction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Graphical Approach to State Variable Selection in Off-policy Learning." pith.science (2026). https://pith.science/paper/XW6L2UAW

@misc{pith2026250100854,
  author       = {Pith},
  title        = {Pith review of: A Graphical Approach to State Variable Selection in Off-policy Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XW6L2UAW}},
  note         = {Machine review of arXiv:2501.00854}
}

read the original abstract

Sequential decision problems are widely studied across many areas of science. A key challenge when learning policies from historical data - a practice commonly referred to as off-policy learning - is how to ``identify'' the impact of a policy of interest when the observed data are not randomized. Off-policy learning has mainly been studied in two settings: dynamic treatment regimes (DTRs), where the focus is on controlling confounding in medical problems with short decision horizons, and offline reinforcement learning (RL), where the focus is on dimension reduction in closed systems such as games. The gap between these two well studied settings has limited the wider application of off-policy learning to many real-world problems. Using the theory for causal inference based on acyclic directed mixed graph (ADMGs), we provide a set of graphical identification criteria in general decision processes that encompass both DTRs and MDPs. We discuss how our results relate to the often implicit causal assumptions made in the DTR and RL literatures and further clarify several common misconceptions. Finally, we present a realistic simulation study for the dynamic pricing problem encountered in container logistics, and demonstrate how violations of our graphical criteria can lead to suboptimal policies.

Figures

Figures reproduced from arXiv: 2501.00854 by the authors.

**Figure 1.** Simple dynamic pricing. Assumption 1 assumes that a variable that is left out in a previous state cannot be part of a new state. Assumption 2 states that the novel variables at time t + 1 must be m-separated from previous state and action variables given the current state St . It is helpful to think of Assumption 2 as a graphical counterpart of the memorylessness assumption that is common in the MDP literature. Assu… view at source ↗

**Figure 2.** Motivating dynamic pricing examples. New edges compared to Figure [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Mean cumulative rewards and 95% confidence intervals relative to the null [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: d-SWIG examples; “null” means no intervention for the corresponding decision. [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Memorylessness and (7) are satisfied for S2 = {A1, L2}, but Assumption 3 fails. 5.2 Why controlling for immediate confounding is not enough For readers with a background in RL, it may seem odd at first why Assumption 3 requires blocking back-door paths across all time …

**Figure 6.** Figure 6: A Markov decision process represented by a DAG [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

**Figure 7.** Figure 7: An illustration of “phantom confounding” created by latent projection. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

**Figure 8.** Figure 8: A graphical representation of POMDP. recent years POMDPs have received considerable attention as a more reasonable trade-off between model flexibility and structural assumptions (e.g. Guo et al. 2016). In stage t of a POMDP, it is assumed that the decision process can …

**Figure 9.** Figure 9: Dynamic pricing ADMG. The basic scenario [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

[1]

Optimal control of Markov processes with incomplete state infor- mation

˚Astr¨ om, K.J (1965). “Optimal control of Markov processes with incomplete state infor- mation”. In: Journal of Mathematical Analysis and Applications 10.1, pp. 174–205. Barto, Andrew G., Richard S. Sutton, and Charles W. Anderson (1983). “Neuronlike adaptive elements that can solve difficult learning control problems”. In: IEEE Trans- actions on Systems...

work page 1965
[76]

Constructing dynamic treat- ment regimes over indefinite time horizons

English. 2013th edition. New York, NY: Springer. Ertefaie, Ashkan and Robert L Strawderman (Sept. 2018). “Constructing dynamic treat- ment regimes over indefinite time horizons”. In: Biometrika 105.4, pp. 963–977. Guo, F. Richard, Anton Rask Lundborg, and Qingyuan Zhao (Sept. 2023). Confounder Selection: Objectives and Approaches. en. Guo, F. Richard and ...

work page arXiv 2016
[311]

Off-policy evaluation in partially observed Markov decision processes under sequential ignorability

Howard, R. A. (1960). Dynamic Programming and Markov Processes . Cambridge, MA: MIT Press. Hu, Yuchen and Stefan Wager (2023). “Off-policy evaluation in partially observed Markov decision processes under sequential ignorability”. In: The Annals of Statistics 51.4, pp. 1561–1585. Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra (1998). ...

work page 1960
[688]

Markov Properties for Acyclic Directed Mixed Graphs

— (2009). Causality: Models, reasoning, and inference . 2nd ed. New York: Cambridge University Press. Puterman, Martin L. (Aug. 2014). Markov Decision Processes: Discrete Stochastic Dy- namic Programming. de. John Wiley & Sons. Richardson, Thomas (2003). “Markov Properties for Acyclic Directed Mixed Graphs”. In: Scandinavian Journal of Statistics 30.1, pp...

arXiv 2009

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Optimal control of Markov processes with incomplete state infor- mation

˚Astr¨ om, K.J (1965). “Optimal control of Markov processes with incomplete state infor- mation”. In: Journal of Mathematical Analysis and Applications 10.1, pp. 174–205. Barto, Andrew G., Richard S. Sutton, and Charles W. Anderson (1983). “Neuronlike adaptive elements that can solve difficult learning control problems”. In: IEEE Trans- actions on Systems...

work page 1965

[2] [76]

Constructing dynamic treat- ment regimes over indefinite time horizons

English. 2013th edition. New York, NY: Springer. Ertefaie, Ashkan and Robert L Strawderman (Sept. 2018). “Constructing dynamic treat- ment regimes over indefinite time horizons”. In: Biometrika 105.4, pp. 963–977. Guo, F. Richard, Anton Rask Lundborg, and Qingyuan Zhao (Sept. 2023). Confounder Selection: Objectives and Approaches. en. Guo, F. Richard and ...

work page arXiv 2016

[3] [311]

Off-policy evaluation in partially observed Markov decision processes under sequential ignorability

Howard, R. A. (1960). Dynamic Programming and Markov Processes . Cambridge, MA: MIT Press. Hu, Yuchen and Stefan Wager (2023). “Off-policy evaluation in partially observed Markov decision processes under sequential ignorability”. In: The Annals of Statistics 51.4, pp. 1561–1585. Kaelbling, Leslie Pack, Michael L. Littman, and Anthony R. Cassandra (1998). ...

work page 1960

[4] [688]

Markov Properties for Acyclic Directed Mixed Graphs

— (2009). Causality: Models, reasoning, and inference . 2nd ed. New York: Cambridge University Press. Puterman, Martin L. (Aug. 2014). Markov Decision Processes: Discrete Stochastic Dy- namic Programming. de. John Wiley & Sons. Richardson, Thomas (2003). “Markov Properties for Acyclic Directed Mixed Graphs”. In: Scandinavian Journal of Statistics 30.1, pp...

arXiv 2009