Pith. sign in

REVIEW 3 major objections 5 minor 95 references

Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Estimating the behavior policy from longer histories provably lowers the asymptotic variance of importance-sampling estimators in off-policy evaluation, while increasing finite-sample bias.

desk verdict First credible theory for the history-dependent behavior-policy paradox in OPE, but the main monotone variance theorem only holds for fixed history length k, and Section 7's data-dependent k selection is not covered by the proof. read the letter →

arxiv 2505.22492 v1 pith:E7WIJF4I submitted 2025-05-28 cs.LG cs.AIstat.ML

Hongyi Zhou , Josiah P. Hanna , Jin Zhu , Ying Yang , Chengchun Shi This is my paper
classification cs.LGcs.AIstat.ML
keywords off-policyevaluationimportancesamplingbehaviorpolicyestimationhistory-dependentpoliciesbias-variancedecompositionasymptoticvariancemarginalizeddoublyrobust
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to explain a known paradox in reinforcement learning: off-policy evaluation via importance sampling can be made more accurate by estimating the behavior policy from the logged data, and by letting that estimate depend on more of the past, even when the true behavior policy depends only on the current state. It proves a bias-variance decomposition for ordinary and sequential importance sampling showing that conditioning the estimated policy on a longer history strictly reduces the asymptotic variance while inflating finite-sample bias. The same mechanism shows that a doubly robust estimator with a correctly specified Q-function gains nothing asymptotically from longer history, while a marginalized importance-sampling estimator loses efficiency as history grows. These results convert an empirical curiosity into a quantitative trade-off: longer history helps as the sample grows, hurts in small samples and long horizons, and the best history length depends on both.

What carries the argument

The load-bearing mechanism is the score-projection expansion of the estimated importance ratio. Replacing the true behavior policy with a likelihood-based estimate changes the estimator by a first-order term equal to minus the orthogonal projection of the oracle IS estimator onto the tangent space spanned by the fitted policy's score; projecting off this term is what removes variance. The space $\mathcal{T}(k)$ is the set of mean-zero random variables orthogonal to that score tangent space, and the leading variance term is the variance of $\operatorname{Proj}_{\mathcal{T}(k)}(\lambda_T G_T)$. For marginalized IS, the central object is the conditional ratio $w_t(k)=\mathbb{E}[\lambda_t\mid H_{t-k:t},A_t]$, and a conditional-variance/Jensen argument shows its variance grows with $k$.

What would settle it

Run a tabular two- or three-step MDP with a known Markov behavior policy, estimate it with histories of length $k=0,1,2$ on a sequence of growing sample sizes, and measure the empirical variance of the ordinary IS estimate across many trials; observing variance at $k=2$ above variance at $k=1$ for large $n$, or MSE that does not settle toward the Theorem 2 form, would falsify the monotone projection claim. Letting $k$ grow with $n$ (for example $k\approx n^{1/4}$) and testing monotonicity would directly probe the fixed-$k$ assumption.

Watch

Extended reading notes

Core claim

At the center is a projection identity. If the behavior policy is fit by maximum likelihood from $k$ steps of history, the first-order effect of replacing the oracle ratio $\lambda_T$ by its estimate is to subtract the projection of the estimator onto the space spanned by the score function of the fitted policy; the MSE then becomes $$\operatorname{MSE}(\hat v_{\mathrm{OIS}}(k)) = \frac{1}{n}\operatorname{Var}\big(\operatorname{Proj}_{\mathcal{T}(k)}(\lambda_T G_T)\big) + \mathcal{O}\left(\frac{(k+1)$C^{{2T}}$R_{\max}^2}{$n^{{3/2}}$\$varepsilon^{2}$}\right).$$ Because the projection spaces nest as $k$ grows, the projected variance is monotonically nonincreasing in $k$, so in large samples longer history is never worse for ordinary and sequential IS. For the doubly robust estimator the same projection acts on the Bellman residual $U_t$; when the Q-function is correct the projected variance is constant in $k$, and when Q is misspecified it falls. Marginalized IS behaves oppositely: its history-dependent ratio $w_t(k)=\mathbb{E}[\lambda_t\mid H_{t-k:t},A_t]$ has variance that increases with $k$, so longer history raises asymptotic MSE.

Load-bearing premise

The result assumes the behavior-policy model is smooth and rich enough to contain the true policy, and that the history length stays fixed while the sample size grows, so the linear-expansion term dominates; if history grows with the data or the model class is too complex, the bias remainder can overtake the variance reduction and monotonicity may fail.

Editorial extensions

If this is right

  • For ordinary and sequential importance sampling, a longer fitting history never increases asymptotic MSE, so the empirical preference for history-dependent behavior policies is justified at large sample sizes.
  • The finite-sample bias of history-dependent estimators is bounded by a term that grows with the horizon $T$ through $C^{2T}$; in small samples or long horizons the bias can outweigh the variance reduction.
  • Doubly robust estimation with a correct Q-function is already asymptotically efficient in this setting; history-dependent behavior-policy ratios neither help nor hurt its asymptotic variance, and only matter under misspecification.
  • Marginalized importance sampling is the exception: conditioning its ratio on more history increases asymptotic MSE, so the 'history helps' rule cannot be applied uniformly across IS variants.
  • Nonparametric sieve estimation of the behavior policy preserves the variance-reduction guarantee even without exact realizability, provided the sieve approximation error decays faster than $n^{-1/4}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the score-projection view suggests a recipe for deciding when history helps—any estimator whose influence function sits in the orthogonal complement of a richer score space should benefit, so the result likely transfers to off-policy learning gradients and variance-reduced policy-gradient updates, not just value estimation.
  • Beyond the paper: the BIC-style history-selection rule proposed in the discussion is plausible, but the paper says its supporting simulations are not reported; treating the rule as validated would require an experiment the decomposition itself does not supply.
  • Beyond the paper: because the bias remainder scales with $C^{2T}$, the 'longer history is better at large n' advice is horizon-sensitive; in long-horizon problems the sample size needed for the variance term to dominate may be exponentially large, so practitioners should expect the crossover point to shift with $T$.
  • Beyond the paper: the MIS result suggests a testable inversion—replacing a history-dependent MIS ratio with a coarser conditioning set will lower variance, which could be used as a diagnostic for whether a fitted MIS ratio is overconditioned in a given dataset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies off-policy evaluation when the behavior policy is estimated from logged trajectories, focusing on why conditioning the estimated behavior policy on a longer history can reduce the MSE of importance-sampling estimators even when the true behavior policy is Markovian. The authors derive a bias-variance decomposition for ordinary IS (OIS), sequential IS (SIS), doubly robust (DR), and marginalized IS (MIS) estimators, with the behavior policy estimated parametrically and nonparametrically. Their central result (Theorem 2) states that the MSE of the OIS estimator equals (1/n) Var(Proj_{T(k)}(λ_T G_T)) plus a remainder of order O((k+1) C^{2T} R_max^2 /(n^{3/2} ε^2)), and that the projection variance decreases monotonically with history length k. They conclude that longer history lowers asymptotic variance for OIS and SIS, leaves correctly specified DR unchanged, improves misspecified DR, and worsens MIS; they also report numerical experiments supporting these conclusions.

Significance. If the central claim holds, the paper gives a principled explanation for a well-documented empirical paradox in OPE, and it provides a unified treatment across four estimator families and two estimation regimes. The projection interpretation in Theorem 2 is a genuinely useful way to view the variance reduction from estimating the behavior policy, and the extension to SIS/DR/MIS substantially broadens the scope of earlier analyses that focused on OIS. The paper also ships detailed appendices with the OIS derivation and numerical validation in CartPole and MuJoCo environments. The main caveat is that the proof is asymptotic in n and treats the history length k as fixed; the practical selection rule in Section 7 allows k to grow with n, and the paper does not establish that the monotonicity claim survives in that regime. The bias claim is also stronger than what the proof supports.

major comments (3)
  1. [§4.1, Theorem 2 and Appendix C.2, Eqs. (12)–(18)] The central monotonicity result is proved only for fixed history length k. The MLE expansion in Eq. (13) uses a fixed-dimensional parameter space, and the remainder in Eq. (18) is of order O((k+1) C^{2T} R_max^2/(n^{3/2} ε^2)), which is o(n^{-1}) only when k is fixed or grows more slowly than √n. The theorem is stated without this qualifier, and Section 7 proposes a data-dependent h* = argmin_h [2n cVar(h) - h log n] that can grow with n. If k_n ~ √n, the remainder is O(n^{-1}), the same order as the variance term, so the projection term is no longer the leading term and the monotonicity conclusion does not follow from the proof. Please state the fixed-k assumption explicitly or provide an argument that handles growing k.
  2. [§4.1 and abstract] The abstract and Section 4.1 claim that history-dependent behavior policy estimation 'increases finite-sample biases.' The proof in Appendix C.2 only establishes the upper bound Bias(bvOIS(k)) = O((k+1) C^T R_max/(n ε^2)); it does not show that the actual bias is monotone in k. The same upper-bound logic is used for SIS and DR. As stated, this is an overclaim: an upper bound that grows linearly in k is not a proof that the bias increases with k. Please either prove monotonicity of the actual bias or soften the claim to indicate that the bias is at most of this order and may increase.
  3. [§4.2, Theorem 4] Theorem 4 is a central result for SIS, but its proof is omitted with the statement that it 'simply follows the proof of Theorem 6 by taking Q(s,a) ≡ 0.' This is not sufficient for a load-bearing theorem in a journal submission. Please supply the proof or give a precise reduction that spells out why the projection term and the remainder bound for SIS are exactly those claimed, including the monotonicity assertion for the variance term.
minor comments (5)
  1. [§7] The notation cVar(h) in the proposed history-length selection rule is undefined in the main text; please define it precisely (e.g., as an estimated sampling variance of the estimator at history length h) before the equation.
  2. [Corollary 7 and surrounding text] There is a typo: 'The equation holds when the Q-function is correctly specified' should read 'The equality holds...'; also, the reference to 'Theorem 5' should be to 'Theorem 6.'
  3. [Appendix C.2, Eq. (14)] In the line before Eq. (14), 'bvOIS - bvOIS' appears to be missing the subscript/superscript; it should read 'bvOIS(k) - bv†OIS'.
  4. [§4.2, after Theorem 4] The sentence 'In addition, the first term on the RHS of (2) is non-decreasing with respect to k' should refer to Eq. (4), not Eq. (2).
  5. [§6, Figure 2 caption] The caption says 'MIS (top panel)' but the figure appears to have three rows with MIS as the bottom panel; please correct the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MSE decompositions are derived from explicit MLE expansions and standard semiparametric projection arguments, not fitted to the conclusions.

full rationale

The paper's central claims are self-contained derivations rather than renamed inputs. Theorem 2 obtains MSE(bvOIS(k)) = n^{-1}Var(Proj_{T(k)}(lambda_T G_T)) + O((k+1)C^{2T}R_max^2/(n^{3/2}epsilon^2)) by Taylor-expanding the estimated-ratio objective around theta*, invoking the standard MLE expansion (equation (13)), and bounding the remainder and cross-covariance terms with Cauchy-Schwarz. No parameter is fitted to the theorem's conclusion, and the error bound is derived from the stated coverage and boundedness assumptions. The monotone variance statement in equation (3) follows from the nested policy classes in Assumption 1 and the Pythagorean theorem for projections onto nested subspaces; although this step is a direct mathematical consequence of the projection construction, it is not a case of defining the target quantity in terms of itself. The DR and MIS results are similarly derived from the Bellman equation (equation (22)) and Jensen's inequality (equation (27)), respectively, rather than from the desired monotonicity. Citations to Hanna et al. (2021) and Kallus & Uehara (2020) are used as motivation, comparison, or as standard semiparametric references for steps such as Neyman orthogonality; these are not load-bearing self-citations that replace an independent proof. Several auxiliary proofs are delegated to standard references or omitted as similar, which is a completeness concern, but not a circularity concern. The fixed-k asymptotic scope, where the remainder is only o(n^{-1}) for k fixed or slowly growing, is a correctness/robustness caveat, not evidence that the theorem reduces to its assumptions by construction.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The theory rests on standard MLE and semiparametric regularity assumptions plus domain assumptions about policy-class nesting, realizability, coverage, bounded rewards, and nuisance convergence. No numbers are fitted to data in the theoretical development, and no new physical or statistical entities are introduced.

assumptions (8)
  • domain assumption Assumption 1: policy classes are nested, Π0 ⊆ Π1 ⊆ Π2 ...
    The paper assumes nesting of policy classes in Section 4. This makes tangent spaces grow with k and drives the Pythagorean projection argument for variance monotonicity. It is plausible for logistic and neural classes but is a structural modeling choice.
  • domain assumption Assumption 2: the true Markov behavior policy is realizable in Π0
    The MLE then converges to πb; without realizability, the score projection is computed at a misspecified parameter and the variance identity does not follow. The paper relaxes this in the sieve setting of Section 5.
  • domain assumption Assumption 3: rewards are bounded by Rmax almost surely
    Bounded rewards are used throughout the remainder bounds in the proofs; unbounded or heavy-tailed rewards are outside the scope of the theorems.
  • domain assumption Assumption 4: all policies are lower bounded by ε and πe/πθ ≤ C
    Coverage is needed to control the importance weights and appears in the C^{2T} factors in the bias and remainder bounds. If coverage fails, the importance weights can blow up.
  • standard math Assumptions 5 and 6: differentiability of policies and non-singular Fisher information
    These are standard MLE regularity conditions used in the √n expansion of bθn around θ* in equation (13).
  • domain assumption Assumption 7: the Bellman residual Ut = Rt - Qt(St,At) + γQt+1(St+1,πe) is bounded by Umax
    This bounds the remainder for the DR estimator in Theorem 6. It is expected to be milder than bounded rewards when Q is well specified.
  • domain assumption For the MIS analysis: estimated weights and Q functions converge with RMSE oP(n^{-1/4}) and satisfy Neyman orthogonality
    In the proof of Theorem 8, the paper replaces estimated weights and Q functions by their oracle limits. This convergence is stated in the appendix rather than as a numbered assumption in the main text.
  • standard math Assumptions 8-13: sieve approximation error o(n^{-1/4}), P-Donsker classes, and existence of a least favorable direction
    These are standard empirical process and semiparametric conditions used in Theorem 9 for nonparametric sieve estimation. They are strong but conventional.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation." pith.science (2026). https://pith.science/paper/E7WIJF4I

@misc{pith2026250522492,
  author       = {Pith},
  title        = {Pith review of: Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7WIJF4I}},
  note         = {Machine review of arXiv:2505.22492}
}
read the original abstract

This paper studies off-policy evaluation (OPE) in reinforcement learning with a focus on behavior policy estimation for importance sampling. Prior work has shown empirically that estimating a history-dependent behavior policy can lead to lower mean squared error (MSE) even when the true behavior policy is Markovian. However, the question of why the use of history should lower MSE remains open. In this paper, we theoretically demystify this paradox by deriving a bias-variance decomposition of the MSE of ordinary importance sampling (IS) estimators, demonstrating that history-dependent behavior policy estimation decreases their asymptotic variances while increasing their finite-sample biases. Additionally, as the estimated behavior policy conditions on a longer history, we show a consistent decrease in variance. We extend these findings to a range of other OPE estimators, including the sequential IS estimator, the doubly robust estimator and the marginalized IS estimator, with the behavior policy estimated either parametrically or non-parametrically.

Figures

Figures reproduced from arXiv: 2505.22492 by the authors.

Figure 1
Figure 1. The left panel is log absolute bias of the three IS estima￾tors. The right panel shows log MSE of three different estimators. Results are averaged over 104 trials. 1. When πb is known to us, the first estimator uses the oracle IS ratio πe/πb to estimate v(πe), vb † IS = En h πe(A) πb(A) R i , where En denotes the empirical average over the (S, A, R) triplets in the offline dataset. According to (1), it is immediate … view at source ↗
Figure 2
Figure 2. Absolute bias (left panel) and log MSE (right panel) of three OPE estimators: SIS (top panel), DR (middle panel), MIS (top panel). The results are averaged over 50 simulations. with substantially different dynamics compared to the other two. The detailed results are deferred to Appendix B. 7. Discussion This paper demystifies the paradox concerning the impact of history-dependent behavior policy estimation on IS-typ… view at source ↗
Figure 3
Figure 3. Bias, log variance and log MSE for OIS estimators across three different environments |Bias| Log Var Log MSE SIS DR MIS 500 1000 2000 4000 8000 500 1000 2000 4000 8000 500 1000 2000 4000 8000 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.8 −2.4 −2.0 −1.6 −2.0 −1.5 −1.0 −2.5 −2.0 −1.5 −1.0 −0.5 −3.0 −2.5 −2.0 −1.5 −4 −3 −2 −1 −2.5 −2.0 −1.5 −1.0 −2.5 −2.0 −1.5 −1.0 Sample Size History Length † 0 1 2 4 [PITH_FULL_I… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Bias, log variance and log MSE for OIS,DR and MIS estimators in Swimmer environment 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

95 extracted references · 64 canonical work pages

  1. [1]

    Behnamnia, A., Aminian, G., Aghaei, A., Shi, C., Tan, V. Y. F., and Rabiee, H. R. Log-sum-exponential estimator for off-policy evaluation and learning. In International Conference on Machine Learning. PMLR, 2025

    2025

  2. [2]

    and Kallus, N

    Bennett, A. and Kallus, N. Proximal reinforcement learning: Efficient off-policy evaluation in partially observed markov decision processes. Operations Research, 72 0 (3): 0 1071--1086, 2024

    2024

  3. [3]

    Off-policy evaluation in doubly inhomogeneous environments

    Bian, Z., Shi, C., Qi, Z., and Wang, L. Off-policy evaluation in doubly inhomogeneous environments. Journal of the American Statistical Association, to appear, 2025

    2025

  4. [4]

    More efficient off-policy evaluation through regularized targeted learning

    Bibaut, A., Malenica, I., Vlassis, N., and Van Der Laan, M. More efficient off-policy evaluation through regularized targeted learning. In International Conference on Machine Learning, pp.\ 654--663. PMLR, 2019

    2019

  5. [5]

    Bossens, D. M. and Thomas, P. S. Low variance off-policy evaluation with state-based importance sampling. In 2024 IEEE Conference on Artificial Intelligence (CAI), pp.\ 871--883. IEEE, 2024

    2024

  6. [6]

    Openai gym, 2016

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016. URL https://arxiv.org/abs/1606.01540

    arXiv 2016

  7. [7]

    and Zhou, A

    Bruns-Smith, D. and Zhou, A. Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders. arXiv preprint arXiv:2302.00662, 2023

    work page arXiv 2023

  8. [8]

    Structured Difference-of-Q via Orthogonal Learning

    Cao, D. and Zhou, A. Orthogonalized estimation of difference of q -functions. arXiv preprint arXiv:2406.08697, 2024

    work page Pith review arXiv 2024

Show all 95 references
  1. [9]

    and Berger, R

    Casella, G. and Berger, R. Statistical inference. CRC press, 2024

    2024

  2. [10]

    and Li, L

    Chapelle, O. and Li, L. An empirical evaluation of thompson sampling. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS'11, pp.\ 2249–2257, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993

    2011

  3. [11]

    and Jiang, N

    Chen, J. and Jiang, N. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp.\ 1042--1051. PMLR, 2019

    2019

  4. [12]

    and Qi, Z

    Chen, X. and Qi, Z. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 3558--3582. PM...

    2022

  5. [13]

    Gaussian approximation of suprema of empirical processes

    Chernozhukov, V., Chetverikov, D., and Kato, K. Gaussian approximation of suprema of empirical processes. The Annals of Statistics, pp.\ 1564--1597, 2014

    2014

  6. [14]

    Double/debiased machine learning for treatment and structural parameters

    Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 0 (1): 0 C1--C68, 01 2018

    2018

  7. [15]

    Coindice: Off-policy confidence interval estimation

    Dai, B., Nachum, O., Chow, Y., Li, L., Szepesvari, C., and Schuurmans, D. Coindice: Off-policy confidence interval estimation. In Advances in Neural Information Processing Systems, volume 33, pp.\ 9398--9411. Curran Associates, Inc., 2020

    2020

  8. [16]

    Doubly Robust Policy Evaluation and Optimization

    Dud \'i k, M., Erhan, D., Langford, J., and Li, L. Doubly Robust Policy Evaluation and Optimization . Statistical Science, 29 0 (4): 0 485 -- 511, 2014. doi:10.1214/14-STS500

    2014 doi

  9. [17]

    A theoretical analysis of deep q-learning

    Fan, J., Wang, Z., Xie, Y., and Yang, Z. A theoretical analysis of deep q-learning. In Learning for dynamics and control, pp.\ 486--489. PMLR, 2020

    2020

  10. [18]

    More robust doubly robust off-policy evaluation

    Farajtabar, M., Chow, Y., and Ghavamzadeh, M. More robust doubly robust off-policy evaluation. ArXiv, abs/1802.03493, 2018

    2018 arXiv

  11. [19]

    Accountable off-policy evaluation with kernel B ellman statistics

    Feng, Y., Ren, T., Tang, Z., and Liu, Q. Accountable off-policy evaluation with kernel B ellman statistics. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 3102--3111. PMLR, 13--18 Jul 2020

    2020

  12. [20]

    Combining parametric and nonparametric models for off-policy evaluation

    Gottesman, O., Liu, Y., Sussex, S., Brunskill, E., and Doshi-Velez, F. Combining parametric and nonparametric models for off-policy evaluation. In International Conference on Machine Learning, pp.\ 2366--2375. PMLR, 2019

    2019

  13. [21]

    D., Thomas, P

    Guo, Z. D., Thomas, P. S., and Brunskill, E. Using options and covariance testing for long horizon off-policy policy evaluation. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 2489–2498, Red Hook, NY, USA, 2017. Curr...

    2017

  14. [22]

    Importance sampling policy evaluation with an estimated behavior policy

    Hanna, J., Niekum, S., and Stone, P. Importance sampling policy evaluation with an estimated behavior policy. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 2605--2613. PMLR, 09--15 Jun 2019

    2019

  15. [23]

    P., Thomas, P

    Hanna, J. P., Thomas, P. S., Stone, P., and Niekum, S. Data-efficient policy evaluation through behavior policy search. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Resea...

    2017

  16. [24]

    P., Niekum, S., and Stone, P

    Hanna, J. P., Niekum, S., and Stone, P. Importance sampling in reinforcement learning with an estimated behavior policy. Mach. Learn., 110 0 (6): 0 1267–1317, 2021. ISSN 0885-6125

    2021

  17. [25]

    Bootstrapping fitted q-evaluation for off-policy inference

    Hao, B., Ji, X., Duan, Y., Lu, H., Szepesvari, C., and Wang, M. Bootstrapping fitted q-evaluation for off-policy inference. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 4074--4084. PMLR, 2021

    2021

  18. [26]

    Importance sampling via the estimated sampler

    Henmi, M., Yoshida, R., and Eguchi, S. Importance sampling via the estimated sampler. Biometrika, 94 0 (4): 0 985--991, 12 2007

    2007

  19. [27]

    W., and Ridder, G

    Hirano, K., Imbens, G. W., and Ridder, G. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71 0 (4): 0 1161--1189, 2003

    2003

  20. [28]

    and Wager, S

    Hu, Y. and Wager, S. Off-policy evaluation in partially observed Markov decision processes under sequential ignorability . The Annals of Statistics, 51 0 (4): 0 1561 -- 1585, 2023. doi:10.1214/23-AOS2287

    2023 doi

  21. [29]

    and Li, L

    Jiang, N. and Li, L. Doubly robust off-policy value evaluation for reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp.\ 652--661, New York, New York, USA, 20--22 Jun 2016. PMLR

    2016

  22. [30]

    and Uehara, M

    Kallus, N. and Uehara, M. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. Journal of Machine Learning Research, 21 0 (167): 0 1--63, 2020. URL http://jmlr.org/papers/v21/19-827.html

    2020

  23. [31]

    and Uehara, M

    Kallus, N. and Uehara, M. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Oper. Res., 70 0 (6): 0 3282–3302, November 2022. ISSN 0030-364X

    2022

  24. [32]

    and Zhou, A

    Kallus, N. and Zhou, A. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems, 33: 0 22293--22304, 2020

    2020

  25. [33]

    Kosorok, M. R. Introduction to Empirical Processes and Semiparametric Inference. Springer New York, NY, 2008

    2008

  26. [34]

    Batch policy learning under constraints

    Le, H., Voloshin, C., and Yue, Y. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 3703--3712. PMLR, 09--15 Jun 2019

    2019

  27. [35]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020

    2005 arXiv

  28. [36]

    Sharp high-probability sample complexities for policy evaluation with linear function approximation

    Li, G., Wu, W., Chi, Y., Ma, C., Rinaldo, A., and Wei, Y. Sharp high-probability sample complexities for policy evaluation with linear function approximation. arXiv preprint arXiv:2305.19001, 2023

    2023 arXiv

  29. [37]

    Off-policy estimation of long-term average outcomes with applications to mobile health

    Liao, P., Klasnja, P., and Murphy, S. Off-policy estimation of long-term average outcomes with applications to mobile health. Journal of the American Statistical Association, 116 0 (533): 0 382--391, 2021

    2021

  30. [38]

    Liao, P., Qi, Z., Wan, R., Klasnja, P., and Murphy, S. A. Batch policy learning in average reward Markov decision processes . The Annals of Statistics, 50 0 (6): 0 3364 -- 3387, 2022

    2022

  31. [39]

    Breaking the curse of horizon: infinite-horizon off-policy estimation

    Liu, Q., Li, L., Tang, Z., and Zhou, D. Breaking the curse of horizon: infinite-horizon off-policy estimation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp.\ 5361–5371, Red Hook, NY, USA, 2018. Curran Associates Inc

    2018

  32. [40]

    and Zhang, S

    Liu, S. and Zhang, S. Efficient policy evaluation with offline data informed behavior policy design. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learn...

    2024

  33. [41]

    D., Chen, C., and Zhang, S

    Liu, S. D., Chen, C., and Zhang, S. Doubly optimal policy evaluation for reinforcement learning. arXiv preprint arXiv:2410.02226, 2024

    2024 arXiv

  34. [42]

    Online estimation and inference for robust policy evaluation in reinforcement learning

    Liu, W., Tu, J., Zhang, Y., and Chen, X. Online estimation and inference for robust policy evaluation in reinforcement learning. arXiv preprint arXiv:2310.02581, 2023

    2023 arXiv

  35. [43]

    J., Laber, E

    Luckett, D. J., Laber, E. B., Kahkoska, A. R., David M. Maahs, E. M.-D., and Kosorok, M. R. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, 115 0 (530): 0 692--706, 2020. doi:10.1080/01621459.2018.1537919

    2020

  36. [44]

    P., and Nowak, R

    Mukherjee, S., Hanna, J. P., and Nowak, R. D. Revar: Strengthening policy evaluation via reduced variance sampling. In Cussens, J. and Zhang, K. (eds.), Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine ...

    2022

  37. [45]

    A., van der Laan, M

    Murphy, S. A., van der Laan, M. J., Robins, J. M., and Group, C. P. P. R. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96 0 (456): 0 1410--1423, 2001

    2001

  38. [46]

    Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections

    Nachum, O., Chow, Y., Dai, B., and Li, L. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019

    2019

  39. [47]

    and Jiang, N

    Nair, Y. and Jiang, N. A spectral approach to off-policy evaluation for pomdps. arXiv preprint arXiv:2109.10502, 2021

    2021 arXiv

  40. [48]

    Off-policy policy evaluation for sequential decisions under unobserved confounding

    Namkoong, H., Keramati, R., Yadlowsky, S., and Brunskill, E. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems, 33: 0 18819--18831, 2020

    2020

  41. [49]

    K., Hsieh, F., and Robins, J

    Newey, W. K., Hsieh, F., and Robins, J. Undersmoothing and bias corrected functional estimation. 1998

    1998

  42. [50]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  43. [51]

    S., and Singh, S

    Precup, D., Sutton, R. S., and Singh, S. P. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp.\ 759–766, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072

    2000

  44. [52]

    Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  45. [53]

    Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70 0 (1): 0 41--55, 1983. ISSN 00063444, 14643510

    1983

  46. [54]

    Conditional importance sampling for off-policy learning

    Rowland, M., Harutyunyan, A., Hasselt, H., Borsa, D., Schaul, T., Munos, R., and Dabney, W. Conditional importance sampling for off-policy learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 45--55. PMLR, 2020

    2020

  47. [55]

    G., and Dabney, W

    Rowland, M., Tang, Y., Lyle, C., Munos, R., Bellemare, M. G., and Dabney, W. The statistical benefits of quantile temporal-difference learning for value estimation. In International Conference on Machine Learning, pp.\ 29210--29231. PMLR, 2023

    2023

  48. [56]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  49. [57]

    Estimating the dimension of a model

    Schwarz, G. Estimating the dimension of a model. The annals of statistics, pp.\ 461--464, 1978

    1978

  50. [58]

    and Ben-David, S

    Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  51. [59]

    Mathematical Statistics

    Shao, J. Mathematical Statistics. Springer, New York, 2nd edition, 2003. ISBN 978-0-387-00179-1. doi:10.1007/b98854

    2003 doi

  52. [60]

    On methods of sieves and penalization

    Shen, X. On methods of sieves and penalization. The Annals of Statistics, 25 0 (6): 0 2555--2591, 1997

    1997

  53. [61]

    Deeply-debiased off-policy interval estimation

    Shi, C., Wan, R., Chernozhukov, V., and Song, R. Deeply-debiased off-policy interval estimation. In International conference on machine learning, pp.\ 9580--9591. PMLR, 2021

    2021

  54. [62]

    A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes

    Shi, C., Uehara, M., Huang, J., and Jiang, N. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pp.\ 20057--20094. PMLR, 2022 a

    2022

  55. [63]

    Statistical inference of the value function for reinforcement learning in infinite-horizon settings

    Shi, C., Zhang, S., Lu, W., and Song, R. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 0 (3): 0 765--793, 12 2022 b

    2022

  56. [64]

    Dynamic causal effects evaluation in a/b testing with a reinforcement learning framework

    Shi, C., Wang, X., Luo, S., Zhu, H., Ye, J., and Song, R. Dynamic causal effects evaluation in a/b testing with a reinforcement learning framework. Journal of the American Statistical Association, 118 0 (543): 0 2059--2071, 2023

    2023

  57. [65]

    Off-policy confidence interval estimation with confounded markov decision process

    Shi, C., Zhu, J., Shen, Y., Luo, S., Zhu, H., and Song, R. Off-policy confidence interval estimation with confounded markov decision process. Journal of the American Statistical Association, 119 0 (545): 0 273--284, 2024

    2024

  58. [66]

    Optimal treatment allocation strategies for a/b testing in partially observable time series experiments

    Sun, K., Kong, L., Zhu, H., and Shi, C. Optimal treatment allocation strategies for a/b testing in partially observable time series experiments. arXiv preprint arXiv:2408.05342, 2024

    2024 arXiv

  59. [67]

    S., Szepesv \'a ri, C., and Maei, H

    Sutton, R. S., Szepesv \'a ri, C., and Maei, H. R. A convergent o(n) algorithm for off-policy temporal-difference learning with linear function approximation. Advances in neural information processing systems, 21 0 (21): 0 1609--1616, 2008

    2008

  60. [68]

    Doubly robust bias reduction in infinite horizon off-policy estimation

    Tang, Z., Feng, Y., Li, L., Zhou, D., and Liu, Q. Doubly robust bias reduction in infinite horizon off-policy estimation. In International Conference on Learning Representations, 2020

    2020

  61. [69]

    Off-policy evaluation in partially observable environments

    Tennenholtz, G., Shalit, U., and Mannor, S. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 10276--10283, 2020

    2020

  62. [70]

    Thomas, P. S. and Brunskill, E. Data-efficient off-policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pp.\ 2139–2148. JMLR.org, 2016

    2016

  63. [71]

    S., Theocharous, G., and Ghavamzadeh, M

    Thomas, P. S., Theocharous, G., and Ghavamzadeh, M. High-confidence off-policy evaluation. In AAAI Conference on Artificial Intelligence, 2015

    2015

  64. [72]

    Tsiatis, A. A. Semiparametric Theory and Missing Data. Springer, 2006

    2006

  65. [73]

    Minimax weight and q-function learning for off-policy evaluation

    Uehara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 9659--9668. PMLR, 13--18 Jul 2020

    2020

  66. [74]

    A review of off-policy evaluation in reinforcement learning

    Uehara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355, 2022

    2022 arXiv

  67. [75]

    Future-dependent value-based off-policy evaluation in pomdps

    Uehara, M., Kiyohara, H., Bennett, A., Chernozhukov, V., Jiang, N., Kallus, N., Shi, C., and Sun, W. Future-dependent value-based off-policy evaluation in pomdps. In Advances in Neural Information Processing Systems, volume 36, pp.\ 15991--16008. Curran Associates, Inc., 2023

    2023

  68. [76]

    W., Wellner, J

    Van Der Vaart, A. W., Wellner, J. A., van der Vaart, A. W., and Wellner, J. A. Weak convergence. Springer, 1996

    1996

  69. [77]

    Safe exploration for efficient policy evaluation and comparison

    Wan, R., Kveton, B., and Song, R. Safe exploration for efficient policy evaluation and comparison. In International Conference on Machine Learning, pp.\ 22491--22511. PMLR, 2022

    2022

  70. [78]

    Blessing from human-ai interaction: Super reinforcement learning in confounded environments

    Wang, J., Qi, Z., and Shi, C. Blessing from human-ai interaction: Super reinforcement learning in confounded environments. arXiv preprint arXiv:2209.15448, 2022

    2022 arXiv

  71. [79]

    Wang, J., Qi, Z., and Wong, R. K. W. Projected state-action balancing weights for offline reinforcement learning . The Annals of Statistics, 51 0 (4): 0 1639 -- 1665, 2023

    2023

  72. [80]

    Off-policy evaluation for tabular reinforcement learning with synthetic trajectories

    Wang, W., Li, Y., and Wu, X. Off-policy evaluation for tabular reinforcement learning with synthetic trajectories. Statistics and Computing, 34 0 (1): 0 41, 2024

    2024

  73. [81]

    Unraveling the interplay between carryover effects and reward autocorrelations in switchback experiments

    Wen, Q., Shi, C., Yang, Y., Tang, N., and Zhu, H. Unraveling the interplay between carryover effects and reward autocorrelations in switchback experiments. In International Conference on Machine Learning. PMLR, 2025

    2025

  74. [82]

    Semiparametrically efficient off-policy evaluation in linear markov decision processes

    Xie, C., Yang, W., and Zhang, Z. Semiparametrically efficient off-policy evaluation in linear markov decision processes. In International Conference on Machine Learning, pp.\ 38227--38257. PMLR, 2023

    2023

  75. [83]

    Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling

    Xie, T., Ma, Y., and Wang, Y.-X. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Curran Associates Inc., Red Hook, NY, USA, 2019 a

    2019

  76. [84]

    Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling

    Xie, T., Ma, Y., and Wang, Y.-X. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in neural information processing systems, 32, 2019 b

    2019

  77. [85]

    Quantile off-policy evaluation via deep conditional generative learning

    Xu, Y., Shi, C., Luo, S., Wang, L., and Song, R. Quantile off-policy evaluation via deep conditional generative learning. arXiv preprint arXiv:2212.14466, 2022

    2022 arXiv

  78. [86]

    An instrumental variable approach to confounded off-policy evaluation

    Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. An instrumental variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pp.\ 38848--38880. PMLR, 2023

    2023

  79. [87]

    and Wang, Y.-X

    Yin, M. and Wang, Y.-X. Asymptotically efficient off-policy evaluation for tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 3948--3958. PMLR, 2020

    2020

  80. [88]

    Two-way deconfounder for off-policy evaluation in causal reinforcement learning

    Yu, S., Fang, S., Peng, R., Qi, Z., Zhou, F., and Shi, C. Two-way deconfounder for off-policy evaluation in causal reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 78169--78200, 2024

    2024

  81. [89]

    A., Laber, E

    Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. A robust method for estimating optimal treatment regimes. Biometrics, 68 0 (4): 0 1010--1018, 05 2012. ISSN 0006-341X

    2012

  82. [90]

    A., Laber, E

    Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika, 100 0 (3): 0 681--694, 2013

    2013

  83. [91]

    and Zhang, Y

    Zhao, X. and Zhang, Y. Asymptotic normality of nonparametric m-estimators with applications to hypothesis testing for panel count data. Statistica Sinica, 27: 0 931--950, 2017. URL https://api.semanticscholar.org/CorpusID:54836455

    2017

  84. [92]

    B., and Kosorok, M

    Zhao, Y.-Q., Zeng, D., Laber, E. B., and Kosorok, M. R. New statistical learning methods for estimating optimal dynamic treatment regimes. Journal of the American Statistical Association, 110 0 (510): 0 583--598, 2015

    2015

  85. [93]

    Distributional shift-aware off-policy interval estimation: A unified error quantification framework

    Zhou, W., Li, Y., Zhu, R., and Qu, A. Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprint arXiv:2309.13278, 2023

    2023 arXiv

  86. [94]

    Robust offline reinforcement learning with heavy-tailed rewards

    Zhu, J., Wan, R., Qi, Z., Luo, S., and Shi, C. Robust offline reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics, pp.\ 541--549. PMLR, 2024

    2024

  87. [95]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.