Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Splitting unmeasured confounders into trajectory- and time-specific factors yields a consistent off-policy value estimator with a finite-sample error bound.

desk verdict A genuinely useful two-way deconfounder for OPE, but the consistency claim runs ahead of the theory: Assumption 4 is doing unproved work. read the letter →

arxiv 2412.05783 v1 pith:B6CNVIYS submitted 2024-12-08 cs.LG stat.ML

Shuguang Yu , Shuxing Fang , Ruixin Peng , Zhengling Qi , Fan Zhou , Chengchun Shi This is my paper · ORCID
classification cs.LGstat.ML
keywords off-policyevaluationunmeasuredconfoundingtwo-wayfixedeffectsdeconfoundingneuraltensornetworkmodel-basedreinforcementlearningcausallatentconfounders
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to evaluate a new policy from logged data when hidden confounders influence both actions and outcomes. It proposes a middle-ground assumption, two-way unmeasured confounding: every hidden confounder is either constant within a trajectory or constant across trajectories, so the latent count is N+T instead of N×T. Under this assumption a neural tensor network learns the confounders together with the environment dynamics, and a plug-in model-based estimator evaluates the target policy. The central theoretical claim is a finite-sample bound on the estimator's mean absolute error that vanishes when the transition error, confounder estimation error, and an autocorrelation exponent all shrink appropriately.

What carries the argument

The central object is the two-way unmeasured confounding assumption, $Z_{i,t} = (U_i^\top, W_t^\top)^\top$, which reduces the number of latent variables from $N \times T$ to $N + T$ and makes consistent estimation feasible. The carrying mechanism is the neural tensor network (NTN), whose bilinear tensor products $u_i^\top W^{[k]} w_t$ let the trajectory- and time-specific confounders interact nonlinearly; the same NTN output feeds both the transition network (conditional Gaussian mean and variance) and the actor network (behavior policy), and the joint loss is optimized to yield the estimated embeddings and dynamics. The estimator then plugs the learned embeddings into Monte Carlo rollouts of the target policy, relying on the latents being policy-agnostic.

What would settle it

Simulate the tumor-growth data-generating process with known ground-truth $U_i$ and $W_t$, train the two-way deconfounder, and measure the mean-squared error of the estimated embeddings against the truth; if this error does not shrink as $N$ and $T$ grow, or if the policy-value error stays bounded away from zero while the other conditions of Theorem 1 hold, the consistency claim falls.

Watch

Extended reading notes

Core claim

The paper's core claim is that consistent off-policy evaluation is achievable without no-unmeasured-confounding, Markov, or proxy assumptions, provided the unmeasured confounders have a two-way structure: each confounder is either a trajectory-specific latent factor $U_i$ or a time-specific latent factor $W_t$. The proposed estimator learns $U_i$ and $W_t$ jointly with a Gaussian transition model and the behavior policy through a neural tensor network, then runs Monte Carlo rollouts of the target policy using the learned model. Theorem 1 bounds the mean absolute error by $C T R_{\max} \varepsilon_P + c T R_{\max} \varepsilon_{U,W} + c R_{\max} N^{-1/2} + c R_{\max} T^{\alpha-1}$; the first two terms come from transition and confounder estimation, the last two from sampling variability across trajectories and time. When those estimation errors vanish and $\alpha < 1$, the bound tends to zero, so the estimator is consistent.

Load-bearing premise

The load-bearing premise is that the estimated neural embeddings converge to the true trajectory- and time-specific confounders in mean squared error; the paper assumes this rather than proving it for its own model.

Editorial extensions

If this is right

  • Under two-way unmeasured confounding, off-policy evaluation becomes feasible without external proxies or Markov assumptions, and the number of latent variables grows only as $N + T$.
  • The finite-sample bound shows the estimation error grows linearly in the horizon $T$, which the paper notes is unavoidable in general, while the standard-deviation terms decay with $N$ and $T^{1-\alpha}$.
  • In the simulated dynamic process, tumor growth example, and MIMIC-III data, the estimator achieves lower logarithmic mean squared error and bias than no-unmeasured-confounding, POMDP-type, and one-way deconfounding baselines.
  • Because the two-way confounders are policy-agnostic, the plug-in estimator avoids estimating the distribution of the latent factors, simplifying the Monte Carlo construction.
  • A sensitivity analysis shows the method degrades gracefully as the two-way assumption is mildly violated, but collapses when the violation is complete.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the proof's hardest condition is Assumption 4, that the learned embeddings track the true latents in mean squared error; the paper only proves an analogue for a linear additive fixed-effects model, not for the neural tensor network, so verifying or repairing this step is the natural next test.
  • Editorial: the sensitivity analysis suggests a simple extension: add a residual term that is both trajectory- and time-specific and use its magnitude as a sensitivity parameter, allowing the method to report how much violation of two-way confounding the conclusions tolerate.
  • Editorial: combining the estimated transition model with a double-robust or marginalized-importance correction could remove the linear $T$ dependence under ergodicity, mirroring standard results in off-policy evaluation.
  • Editorial: in the MIMIC-III analysis the policy-ranking check is qualitative; a sharper validation would withhold final outcomes and compare TWD's recommended ranking against a clinical benchmark or an oracle informed by full patient records.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses off-policy evaluation (OPE) in reinforcement learning when unmeasured confounders affect both actions and transitions/rewards. It introduces a 'two-way unmeasured confounding' (TWUC) model in which each latent confounder decomposes into a trajectory-specific component U_i and a time-specific component W_t, and proposes a two-way deconfounder that learns these embeddings together with a transition model and a behavior-policy model using a neural tensor network, followed by Monte Carlo policy evaluation. The theoretical section states a finite-sample error bound (Theorem 1) that is linear in the transition error ε_P and the embedding error ε_{U,W}. Experiments on two synthetic environments and MIMIC-III compare the method against model-based, importance-sampling, doubly robust, and POMDP baselines.

Significance. The TWUC assumption is a genuinely useful middle ground between unconstrained and one-way confounding, and the joint NTN architecture plus the model-based plug-in estimator is a sensible way to instantiate it. The paper ships code, provides sample-splitting proof ideas, and includes a broad set of experiments, including a real-data study; these are real strengths. However, the paper's advertised consistency result is conditional on an unproved embedding-recovery assumption. The theoretical contribution therefore currently establishes a conditional guarantee only, and the empirical evaluation cannot substitute for the missing identifiability/convergence analysis.

major comments (3)
  1. [Section 4, Assumption 4 and Theorem 1] The conclusion that E|η̂π−ηπ| vanishes as N,T grow requires ε_U,W→0 (in fact T ε_U,W→0), but Assumption 4 is never verified for the proposed neural tensor network estimator. The sentence 'According to Proposition 3, these errors are negligible under simple models' does not support this. Proposition 3 (Appendix B.4) bounds the prediction MSE of a linear additive two-way fixed-effects model; it does not bound (1/N)Σ_i E‖Û_i−U_i‖² or (1/T)Σ_t E‖Ŵ_t−W_t‖². Moreover, in that linear model the factors are identified only up to an additive constant, so Euclidean embedding error need not vanish even as predictions improve. For the nonlinear NTN estimator no identifiability condition or consistency theorem for the embeddings is supplied, so the central consistency claim of the abstract rests on an unsupported premise.
  2. [Appendix B.5, Step 1 (bound on I2)] The proof states that 'as neural networks are Lipschitz continuous functions of their parameters, the above total variation norm is proportional to ∥(Ûᵀ_i,Ŵᵀ_t)ᵀ − (Uᵀ_i,Wᵀ_t)ᵀ∥₂' and then applies Assumption 4. This is not a consequence of Lipschitz continuity; it is an additional structural assumption that the learned transition distribution changes in total variation at a uniform rate in the embedding error, with a constant independent of (a,o) and of the estimated network parameters. No such bound or constant is stated. Since this is exactly the step that produces the term cT Rmax ε_U,W, the second term of Theorem 1 is not justified as written.
  3. [Section 4 versus Section 3] Theorem 1 is stated only in a tabular setting with discrete observation, action, and latent spaces, while the proposed estimator in Section 3 is defined for continuous observations and uses a conditional Gaussian NTN; the experiments also use continuous observations. The paper does not provide a discretization or approximation argument showing that the tabular bound applies to the implemented estimator. Thus the theorem, even if Assumptions 1-5 held, would not cover the algorithm as used in the experiments.
minor comments (4)
  1. [Appendix B.4] The displayed formula MSE(Ŷ)=σ²(N+T+2)/(NT) appears to be off by one: the projection matrix X(XᵀX)⁻Xᵀ has rank N+T+1 because the unit and time dummy columns sum to the same all-ones vector. The rate is unaffected.
  2. [Appendix D.4] In the tailored policy for normalized SOFA < 0.7, the text assigns p(A_v=1)=0.2 and then p(A_v=2)=p(A_v=1)=p(A_v=0)=0, which is internally inconsistent; please correct the intended probabilities.
  3. [Appendix C.2] The learning-rate search range is written as [0.005, 0.001], which is in decreasing order and is presumably meant to be [0.001, 0.005] or similar.
  4. [Section 5.4 and Table 1] The text and table refer to one-way deconfounder variants as OWD-NI and OWD-NT, while Appendix C.3 labels them TWD-NI and TWD-NT; please standardize the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the policy-value bound is a conditional plug-in analysis; the latent-embedding error assumption is an unproven premise, not a fitted quantity renamed as a prediction.

full rationale

The derivation chain is self-contained in the sense required by the circularity definition. The target ηπ is defined independently as the expected cumulative reward under the target policy π, and the proposed estimator is a Monte Carlo plug-in that uses the learned transition function bP and the estimated embeddings (Û_i, Ŵ_t) under cross-fitting. Theorem 1 states E|η̂π − ηπ| ≤ C T Rmax ε_P + c T Rmax ε_U,W + c Rmax N^{-1/2} + c Rmax T^{α-1} under Assumptions 1–5; the bound is linear in the assumed transition error ε_P and the assumed embedding error ε_U,W. This is a soundness decomposition, not a construction in which the predicted quantity is defined by the fitted parameters. The only disputable link is the paper's assertion that Assumption 4 is 'negligible under simple models' via Proposition 3: Proposition 3 bounds prediction MSE in a linear additive two-way model, not the Euclidean parameter error of the neural tensor network embeddings, so the advertised consistency rests on an unsupported premise. That is a correctness or soundness gap, not circularity. The self-citations (Shi et al. 2022a, Bian et al. 2023, Zhou et al. 2023b) are used for baselines and data setup, not to prove the central bound. No equation in the paper reduces the target value to the fitted loss, and no fitted parameter is renamed as a prediction. The limitations section also honestly acknowledges that TWUC may fail under trajectory- and time-specific or policy-dependent confounders. Therefore, no circular step is present.

Assumptions & free parameters 3 free parameters · 6 assumptions · 2 invented entities

The method's core is a latent factor model: it assumes the world is governed by N trajectory factors and T time factors, with no cell-specific unobserved variables. It also assumes these factors are estimable with small error by the neural tensor network. Neither identifiability nor convergence of the embeddings is proven for the nonlinear model; the theory takes those as assumptions. Hyperparameters add further tuning choices.

free parameters (3)
  • Embedding dimension d = chosen from {2,4,8} by grid search
    The dimension of the two-way latent confounders is a model choice searched on validation data; the theory treats d as fixed and does not provide a selection rule.
  • Loss weighting alpha = chosen from {0.0, 0.3, 0.5, 0.7}
    Balances transition likelihood and behavior-policy cross-entropy; tuned by grid search. It affects which embeddings are learned.
  • Learning rate, batch size, weight decay = lr in [0.005, 0.001], bs in [2^8, 2^12], weight decay in [0.01, 0.0001]
    Standard training hyperparameters chosen by validation; not part of the theorem but required to reproduce the experiments.
assumptions (6)
  • domain assumption Two-way unmeasured confounding (TWUC): Z_{i,t} = (U_i^T, W_t^T)^T
    The entire method is built on the assumption that all unmeasured confounders are either trajectory-specific or time-specific, not both. Section 2 defines it. Appendix A.2 acknowledges it might not handle cell-specific confounders.
  • domain assumption Policy-agnostic confounders: the two-way confounders are unaffected by actions or policies
    Stated after the DGP in Section 2. Required for the plug-in estimator in equation (1).
  • domain assumption Coverage Assumption 1: C = sup d_pi / d_Dbar < infinity
    Needed for the change-of-measure step in the proof of Theorem 1 in Appendix B.5.
  • domain assumption Error bounds Assumption 3 and Assumption 4: transition and latent confounder estimation errors bounded by epsilon_P and epsilon_U,W
    These are assumed, not derived for the neural tensor network. The theorem's consistency conclusion depends on both tending to zero.
  • ad hoc to paper Lipschitz continuity of the network and proportionality of TV distance to embedding error
    In the proof of term I2 in Appendix B.5, the paper asserts the TV norm is proportional to the embedding difference because neural networks are Lipschitz. This is stated without constants or conditions.
  • standard math Rank of the two-way design matrix is N+T+2 in Proposition 3
    The MSE computation assumes the design matrix has full column rank N+T+2; actually the individual and time dummy columns are collinear, so the rank is at most N+T+1. The conclusion still holds.
invented entities (2)
  • Trajectory-specific latent confounders U_i
    purpose: Capture time-invariant unmeasured factors specific to each trajectory such as a patient's stable baseline; they affect behavior policy and transitions.
    No falsifiable prediction outside the fitted model is provided. Identifiability of U_i is assumed via estimation error bound Assumption 4, not established.
  • Time-specific latent confounders W_t
    purpose: Capture trajectory-invariant time-varying shocks such as weather or hospital-wide conditions that influence all trajectories at time t.
    Same as U_i: no independent measurement or falsifiable handle; the paper suggests a Markov property test as partial evaluation but does not carry it out.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning." pith.science (2026). https://pith.science/paper/B6CNVIYS

@misc{pith2026241205783,
  author       = {Pith},
  title        = {Pith review of: Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6CNVIYS}},
  note         = {Machine review of arXiv:2412.05783}
}
read the original abstract

This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which a model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.

Figures

Figures reproduced from arXiv: 2412.05783 by the authors.

Figure 1
Figure 1. The directed acyclic graphs of data generating processes under different assumptions. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) : An overview of the proposed network architecture. (b) : The upper panel reports MSEs under different unmeasured confounding assumptions for fitting the observed data whereas the bottom panel displays the MSEs for off-policy value prediction. The unconstrained unmeasured confounding model shows the best fit for the training data, due to overfitting. The OPE estimator under the proposed two-way unmeasured confou… view at source ↗
Figure 3
Figure 3. Logarithmic MSE and Bias of various estimators for the simulated dynamic process and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) : The estimated policy value for four target policies in real-world dataset. (b) : Average root MSE and its standard error in the results for predicting immediate reward and next observation. The results are aggregated over 20 runs. is two-dimensional, including th…
Figure 5
Figure 5. Figure 5: Sensitivity analysis for the simulated dynamic process and tumor growth experiment. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

Reference graph

Works this paper leans on

106 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Formulation and estimation of dynamic models using panel data

    Theodore Wilbur Anderson and Cheng Hsiao. Formulation and estimation of dynamic models using panel data. Journal of econometrics, 18 0 (1): 0 47--82, 1982

    1982

  2. [2]

    Doubly robust identification for causal panel data models

    Dmitry Arkhangelsky and Guido W Imbens. Doubly robust identification for causal panel data models. The Econometrics Journal, 25 0 (3): 0 649--674, 2022

    2022

  3. [3]

    Design-based analysis in difference-in-differences settings with staggered adoption

    Susan Athey and Guido W Imbens. Design-based analysis in difference-in-differences settings with staggered adoption. Journal of Econometrics, 226 0 (1): 0 62--79, 2022

    2022

  4. [4]

    Econometric analysis of panel data, volume 4

    Badi Hani Baltagi and Badi H Baltagi. Econometric analysis of panel data, volume 4. Springer, 2008

    2008

  5. [5]

    Genetic risk profiles for cancer susceptibility and therapy response

    Helmut Bartsch, Heike Dally, Odilia Popanda, Angela Risch, and Peter Schmezer. Genetic risk profiles for cancer susceptibility and therapy response. Cancer Prevention, pages 19--36, 2007

    2007

  6. [6]

    Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes

    Andrew Bennett and Nathan Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. Operations Research, 2023

    2023

  7. [7]

    Off-policy Evaluation in Doubly Inhomogeneous Environments

    Zeyu Bian, Chengchun Shi, Zhengling Qi, and Lan Wang. Off-policy evaluation in doubly inhomogeneous environments. arXiv preprint arXiv:2306.08719, 2023

    work page Pith review arXiv 2023

  8. [8]

    Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders

    Ioana Bica, Ahmed Alaa, and Mihaela Van Der Schaar. Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders. In International Conference on Machine Learning, pages 884--895. PMLR, 2020

    2020

Show all 106 references
  1. [9]

    Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders

    David Bruns-Smith and Angela Zhou. Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders. arXiv preprint arXiv:2302.00662, 2023

    2023

  2. [10]

    Treatment effects in interactive fixed effects models with a small number of time periods

    Brantly Callaway and Sonia Karami. Treatment effects in interactive fixed effects models with a small number of time periods. Journal of Econometrics, 233 0 (1): 0 184--208, 2023

    2023

  3. [11]

    Statistical inference

    George Casella and Roger L Berger. Statistical inference. Cengage Learning, 2021

    2021

  4. [12]

    Universal off-policy evaluation

    Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 34: 0 27475--27490, 2021

    2021

  5. [13]

    Testing for the markov property in time series

    Bin Chen and Yongmiao Hong. Testing for the markov property in time series. Econometric Theory, 28 0 (1): 0 130--178, 2012

    2012

  6. [14]

    Information-theoretic considerations in batch reinforcement learning

    Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. CoRR, abs/1905.00360, 2019

    1905 arXiv

  7. [15]

    On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation

    Xiaohong Chen and Zhengling Qi. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. In International Conference on Machine Learning, pages 3558--3582. PMLR, 2022

    2022

  8. [16]

    On instrumental variable regression for deep offline policy evaluation

    Yutian Chen, Liyuan Xu, Caglar Gulcehre, Tom Le Paine, Arthur Gretton, Nando De Freitas, and Arnaud Doucet. On instrumental variable regression for deep offline policy evaluation. Journal of Machine Learning Research, 23 0 (302): 0 1--40, 2022

    2022

  9. [17]

    Double/debiased machine learning for treatment and structural parameters

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 0 (1): 0 C1--C68, 2018

    2018

  10. [18]

    A crash course in good and bad controls

    Carlos Cinelli, Andrew Forney, and Judea Pearl. A crash course in good and bad controls. Sociological Methods & Research, 53: 0 004912412210995, 05 2022. doi:10.1177/00491241221099552

    2022 doi

  11. [19]

    Coindice: Off-policy confidence interval estimation

    Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesv \'a ri, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33: 0 9398--9411, 2020

    2020

  12. [20]

    Comment: Reflections on the deconfounder, 2019

    Alexander D'Amour. Comment: Reflections on the deconfounder, 2019

    2019

  13. [21]

    Two-way fixed effects estimators with heterogeneous treatment effects

    Cl \'e ment De Chaisemartin and Xavier d’Haultfoeuille. Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110 0 (9): 0 2964--2996, 2020

    2020

  14. [22]

    Counterfactual inference for sequential experiments

    Raaz Dwivedi, Katherine Tian, Sabina Tomkins, Predrag Klasnja, Susan Murphy, and Devavrat Shah. Counterfactual inference for sequential experiments. arXiv preprint arXiv:2202.06891, 2022

    2022 arXiv

  15. [23]

    A theoretical analysis of deep q-learning

    Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for dynamics and control, pages 486--489. PMLR, 2020

    2020

  16. [24]

    More robust doubly robust off-policy evaluation

    Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, pages 1447--1456. PMLR, 2018

    2018

  17. [25]

    Deep neural networks for estimation and inference

    Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89 0 (1): 0 181--213, 2021

    2021

  18. [26]

    Non-parametric panel data models with interactive fixed effects

    Joachim Freyberger. Non-parametric panel data models with interactive fixed effects. The Review of Economic Studies, 85 0 (3): 0 1824--1851, 2018

    2018

  19. [27]

    Offline reinforcement learning with instrumental variables in confounded markov decision processes

    Zuyue Fu, Zhengling Qi, Zhaoran Wang, Zhuoran Yang, Yanxun Xu, and Michael R Kosorok. Offline reinforcement learning with instrumental variables in confounded markov decision processes. arXiv preprint arXiv:2209.08666, 2022

    2022 arXiv

  20. [28]

    Prediction of treatment response for combined chemo-and radiation therapy for non-small cell lung cancer patients using a bio-mathematical model

    Changran Geng, Harald Paganetti, and Clemens Grassberger. Prediction of treatment response for combined chemo-and radiation therapy for non-small cell lung cancer patients using a bio-mathematical model. Scientific reports, 7 0 (1): 0 13542, 2017

    2017

  21. [29]

    Issues in assessing the contribution of research and development to productivity growth

    Zvi Griliches. Issues in assessing the contribution of research and development to productivity growth. The bell journal of economics, pages 92--116, 1979

    1979

  22. [30]

    Richard Guo, Anton Rask Lundborg, and Qingyuan Zhao

    F. Richard Guo, Anton Rask Lundborg, and Qingyuan Zhao. Confounder selection: Objectives and approaches, 2023

    2023

  23. [31]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019 a

    1912 arXiv

  24. [32]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555--2565. PMLR, 2019 b

    2019

  25. [33]

    Bootstrapping fitted q-evaluation for off-policy inference

    Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning, pages 4074--4084. PMLR, 2021

    2021

  26. [34]

    Sequential deconfounding for causal inference with unobserved confounders

    Tobias Hatt and Stefan Feuerriegel. Sequential deconfounding for causal inference with unobserved confounders. arXiv preprint arXiv:2104.09323, 2021

    2021 arXiv

  27. [35]

    Neural collaborative filtering

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pages 173--182, 2017

    2017

  28. [36]

    A policy gradient method for confounded pomdps

    Mao Hong, Zhengling Qi, and Yanxun Xu. A policy gradient method for confounded pomdps. arXiv preprint arXiv:2305.17083, 2023

    2023 arXiv

  29. [37]

    Collaborative filtering for implicit feedback datasets

    Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference on data mining, pages 263--272. Ieee, 2008

    2008

  30. [38]

    On the use of two-way fixed effects regression models for causal inference with panel data

    Kosuke Imai and In Song Kim. On the use of two-way fixed effects regression models for causal inference with panel data. Political Analysis, 29 0 (3): 0 405--415, 2021

    2021

  31. [39]

    Off-policy evaluation via off-policy classification

    Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-policy classification. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 5437--5448, 2019

    2019

  32. [40]

    When to trust your model: Model-based policy optimization

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    A survey on knowledge graphs: Representation, acquisition, and applications

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems, 33 0 (2): 0 494--514, 2021

    2021

  34. [42]

    A note on loss functions and error compounding in model-based reinforcement learning

    Nan Jiang. A note on loss functions and error compounding in model-based reinforcement learning. arXiv preprint arXiv:2404.09946, 2024

    2024 arXiv

  35. [43]

    Doubly robust off-policy value evaluation for reinforcement learning

    Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 652--661. PMLR, 2016

    2016

  36. [44]

    Mimic-iii, a freely accessible critical care database

    Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3 0 (1): 0 1--9, 2016

    2016

  37. [45]

    Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning

    Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research, 70 0 (6): 0 3282--3302, 2022

    2022

  38. [46]

    Confounding-robust policy evaluation in infinite-horizon reinforcement learning

    Nathan Kallus and Angela Zhou. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems, 33: 0 22293--22304, 2020

    2020

  39. [47]

    Offline policy evaluation and optimization under confounding

    Chinmaya Kausik, Yangyi Lu, Kevin Tan, Maggie Makar, Yixin Wang, and Ambuj Tewari. Offline policy evaluation and optimization under confounding. arXiv preprint arXiv:2211.16583, 2022

    2022 arXiv

  40. [48]

    Learning mixtures of markov chains and mdps

    Chinmaya Kausik, Kevin Tan, and Ambuj Tewari. Learning mixtures of markov chains and mdps. In International Conference on Machine Learning, pages 15970--16017. PMLR, 2023

    2023

  41. [49]

    Causal reinforcement learning: An instrumental variable approach

    Jin Li, Ye Luo, and Xiaowei Zhang. Causal reinforcement learning: An instrumental variable approach. arXiv preprint arXiv:2103.04021, 2021

    2021 arXiv

  42. [50]

    Instrumental variable value iteration for causal offline reinforcement learning

    Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang, Mladen Kolar, and Zhaoran Wang. Instrumental variable value iteration for causal offline reinforcement learning. arXiv preprint arXiv:2102.09907, 2021 a

    2021 arXiv

  43. [51]

    Off-policy estimation of long-term average outcomes with applications to mobile health

    Peng Liao, Predrag Klasnja, and Susan Murphy. Off-policy estimation of long-term average outcomes with applications to mobile health. Journal of the American Statistical Association, 116 0 (533): 0 382--391, 2021 b

    2021

  44. [52]

    Batch policy learning in average reward markov decision processes

    Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. Annals of statistics, 50 0 (6): 0 3364, 2022

    2022

  45. [53]

    Forecasting treatment responses over time using recurrent marginal structural networks

    Bryan Lim. Forecasting treatment responses over time using recurrent marginal structural networks. Advances in neural information processing systems, 31, 2018

    2018

  46. [54]

    Breaking the curse of horizon: Infinite-horizon off-policy estimation

    Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. Advances in neural information processing systems, 31, 2018

    2018

  47. [55]

    Provably good batch off-policy reinforcement learning without great exploration

    Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume...

    2020

  48. [56]

    Causal effect inference with deep latent-variable models

    Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017

    2017

  49. [57]

    Deconfounding reinforcement learning in observational settings

    Chaochao Lu, Bernhard Sch \"o lkopf, and Jos \'e Miguel Hern \'a ndez-Lobato. Deconfounding reinforcement learning in observational settings. arXiv preprint arXiv:1812.10576, 2018

    2018 arXiv

  50. [58]

    Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes

    Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations, 2023

    2023

  51. [59]

    Estimating causal peer influence in homophilous social networks by inferring latent locations

    Edward McFowland III and Cosma Rohilla Shalizi. Estimating causal peer influence in homophilous social networks by inferring latent locations. Journal of the American Statistical Association, 118 0 (541): 0 707--718, 2023

    2023

  52. [60]

    Off-policy evaluation for episodic partially observable markov decision processes under non-parametric models

    Rui Miao, Zhengling Qi, and Xiaoke Zhang. Off-policy evaluation for episodic partially observable markov decision processes under non-parametric models. Advances in Neural Information Processing Systems, 35: 0 593--606, 2022

    2022

  53. [61]

    Empirical production function free of management bias

    Yair Mundlak. Empirical production function free of management bias. Journal of Farm Economics, 43 0 (1): 0 44--56, 1961

    1961

  54. [62]

    A spectral approach to off-policy evaluation for pomdps

    Yash Nair and Nan Jiang. A spectral approach to off-policy evaluation for pomdps. arXiv preprint arXiv:2109.10502, 2021

    2021 arXiv

  55. [63]

    Off-policy policy evaluation for sequential decisions under unobserved confounding

    Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33: 0 18819--18831, 2020

    2020

  56. [64]

    A novel embedding model for knowledge base completion based on convolutional neural network

    Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based on convolutional neural network. arXiv preprint arXiv:1712.02121, 2017

    2017 arXiv

  57. [65]

    A review of relational machine learning for knowledge graphs

    Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104 0 (1): 0 11--33, 2015

    2015

  58. [66]

    Automated cars meet human drivers: responsible human-robot coordination and the ethics of mixed traffic

    Sven Nyholm and Jilles Smids. Automated cars meet human drivers: responsible human-robot coordination and the ethics of mixed traffic. Ethics and Information Technology, 22: 0 335--344, 2020

    2020

  59. [67]

    blessings of multiple causes

    Elizabeth L. Ogburn, Ilya Shpitser, and Eric J.Tchetgen Tchetgen. Comment on “blessings of multiple causes”. Journal of the American Statistical Association, 114 0 (528): 0 1611--1615, October 2019. ISSN 0162-1459. doi:10.1080/01621459.2019.1689139

    2019

  60. [68]

    the blessings of multiple causes

    Elizabeth L Ogburn, Ilya Shpitser, and Eric J Tchetgen Tchetgen. Counterexamples to" the blessings of multiple causes" by wang and blei. arXiv preprint arXiv:2001.06555, 2020

    2001 arXiv

  61. [69]

    A critical look at the consistency of causal estimation with deep latent variable models

    Severi Rissanen and Pekka Marttinen. A critical look at the consistency of causal estimation with deep latent variable models. Advances in Neural Information Processing Systems, 34: 0 4207--4217, 2021

    2021

  62. [70]

    Doubly robust difference-in-differences estimators

    Pedro HC Sant’Anna and Jun Zhao. Doubly robust difference-in-differences estimators. Journal of Econometrics, 219 0 (1): 0 101--122, 2020

    2020

  63. [71]

    Importance resampling for off-policy prediction

    Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance resampling for off-policy prediction. Advances in Neural Information Processing Systems, 32, 2019

    2019

  64. [72]

    Nonparametric regression using deep neural networks with relu activation function

    Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48 0 (4), August 2020. ISSN 0090-5364. doi:10.1214/19-aos1875

    2020 doi

  65. [73]

    On counterfactual inference with unobserved confounding

    Abhin Shah, Raaz Dwivedi, Devavrat Shah, and Gregory W Wornell. On counterfactual inference with unobserved confounding. arXiv preprint arXiv:2211.08209, 2022

    2022 arXiv

  66. [74]

    Does the markov decision process fit the data: Testing for the markov property in sequential decision making

    Chengchun Shi, Runzhe Wan, Rui Song, Wenbin Lu, and Ling Leng. Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In International Conference on Machine Learning, pages 8807--8817. PMLR, 2020

    2020

  67. [75]

    Deeply-debiased off-policy interval estimation

    Chengchun Shi, Runzhe Wan, Victor Chernozhukov, and Rui Song. Deeply-debiased off-policy interval estimation. In International conference on machine learning, pages 9580--9591. PMLR, 2021

    2021

  68. [76]

    A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes

    Chengchun Shi, Masatoshi Uehara, Jiawei Huang, and Nan Jiang. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pages 20057--20094. PMLR, 2022 a

    2022

  69. [77]

    Statistical inference of the value function for reinforcement learning in infinite-horizon settings

    Chengchun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 0 (3): 0 765--793, 2022 b

    2022

  70. [78]

    Off-policy confidence interval estimation with confounded markov decision process

    Chengchun Shi, Jin Zhu, Shen Ye, Shikai Luo, Hongtu Zhu, and Rui Song. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, pages 1--12, 2022 c

    2022

  71. [79]

    Mediation pathway selection with unmeasured mediator-outcome confounding

    Kang Shuai, LAn Liu, Yangbo He, and Wei Li. Mediation pathway selection with unmeasured mediator-outcome confounding. arXiv preprint arXiv:2311.16793, 2023

    2023

  72. [80]

    Reasoning with neural tensor networks for knowledge base completion

    Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013

    2013

  73. [81]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  74. [82]

    Doubly robust bias reduction in infinite horizon off-policy estimation

    Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off-policy estimation. arXiv preprint arXiv:1910.07186, 2019

    1910 arXiv

  75. [83]

    An introduction to proximal causal learning

    Eric J Tchetgen Tchetgen, Andrew Ying, Yifan Cui, Xu Shi, and Wang Miao. An introduction to proximal causal learning. arXiv preprint arXiv:2009.10982, 2020

    2009 arXiv

  76. [84]

    Off-policy evaluation in partially observable environments

    Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276--10283, 2020

    2020

  77. [85]

    Data-efficient off-policy policy evaluation for reinforcement learning

    Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139--2148. PMLR, 2016

    2016

  78. [86]

    High-confidence off-policy evaluation

    Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  79. [87]

    Implicit causal models for genome-wide association studies

    Dustin Tran and David M Blei. Implicit causal models for genome-wide association studies. arXiv preprint arXiv:1710.10742, 2017

    2017 arXiv

  80. [88]

    Pessimistic model-based offline RL: PAC bounds and posterior sampling under partial coverage

    Masatoshi Uehara and Wen Sun. Pessimistic model-based offline RL: PAC bounds and posterior sampling under partial coverage. CoRR, abs/2107.06226, 2021

    2021 arXiv

  81. [89]

    Minimax weight and q-function learning for off-policy evaluation

    Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning, pages 9659--9668. PMLR, 2020

    2020

  82. [90]

    Using embeddings to correct for unobserved confounding in networks

    Victor Veitch, Yixin Wang, and David Blei. Using embeddings to correct for unobserved confounding in networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  83. [91]

    Adapting text embeddings for causal inference

    Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference. In Conference on Uncertainty in Artificial Intelligence, pages 919--928. PMLR, 2020

    2020

  84. [92]

    Relational deep learning: A deep latent variable model for link prediction

    Hao Wang, Xingjian Shi, and Dit-Yan Yeung. Relational deep learning: A deep latent variable model for link prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017

    2017

  85. [93]

    Blessing from experts: Super reinforcement learning in confounded environments

    Jiayi Wang, Zhengling Qi, and Chengchun Shi. Blessing from experts: Super reinforcement learning in confounded environments. arXiv preprint arXiv:2209.15448, 2022

    2022 arXiv

  86. [94]

    Provably efficient causal reinforcement learning with confounded observational data

    Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. Advances in Neural Information Processing Systems, 34: 0 21164--21175, 2021

    2021

  87. [95]

    The blessings of multiple causes

    Yixin Wang and David M Blei. The blessings of multiple causes. Journal of the American Statistical Association, 114 0 (528): 0 1574--1596, 2019

    2019

  88. [96]

    The deconfounded recommender: A causal inference approach to recommendation

    Yixin Wang, Dawen Liang, Laurent Charlin, and David M Blei. The deconfounded recommender: A causal inference approach to recommendation. arXiv preprint arXiv:1808.06581, 2018

    2018 arXiv

  89. [97]

    Semiparametrically efficient off-policy evaluation in linear markov decision processes

    Chuhan Xie, Wenhao Yang, and Zhihua Zhang. Semiparametrically efficient off-policy evaluation in linear markov decision processes. In International Conference on Machine Learning, pages 38227--38257. PMLR, 2023

    2023

  90. [98]

    Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling

    Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems, 32, 2019

    2019

  91. [99]

    An instrumental variable approach to confounded off-policy evaluation

    Yang Xu, Jin Zhu, Chengchun Shi, Shikai Luo, and Rui Song. An instrumental variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pages 38848--38880. PMLR, 2023

    2023

  92. [100]

    Strategic decision-making in the presence of information asymmetry: Provably efficient rl with algorithmic instruments

    Mengxin Yu, Zhuoran Yang, and Jianqing Fan. Strategic decision-making in the presence of information asymmetry: Provably efficient rl with algorithmic instruments. arXiv preprint arXiv:2208.11040, 2022

    2022 arXiv

  93. [101]

    Mopo: Model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 0 14129--14142, 2020

    2020

  94. [102]

    Markov decision processes with unobserved confounders: A causal approach

    Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical report, Technical report, Technical Report R-23, Purdue AI Lab, 2016

    2016

  95. [103]

    The medical deconfounder: assessing treatment effects with electronic health records

    Linying Zhang, Yixin Wang, Anna Ostropolets, Jami J Mulgrave, David M Blei, and George Hripcsak. The medical deconfounder: assessing treatment effects with electronic health records. In Machine Learning for Healthcare Conference, pages 490--512. PMLR, 2019

    2019

  96. [104]

    Distributional shift-aware off-policy interval estimation: A unified error quantification framework

    Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu. Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprint arXiv:2309.13278, 2023 a

    2023 arXiv

  97. [105]

    Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach

    Yunzhe Zhou, Zhengling Qi, Chengchun Shi, and Lexin Li. Optimizing pessimism in dynamic treatment regimes: A bayesian learning approach. In International Conference on Artificial Intelligence and Statistics, pages 6704--6721. PMLR, 2023 b

    2023

  98. [106]

    Testing for the markov property in time series via deep conditional generative learning

    Yunzhe Zhou, Chengchun Shi, Lexin Li, and Qiwei Yao. Testing for the markov property in time series via deep conditional generative learning. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85 0 (4): 0 1204--1222, 2023 c

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.