Pith. sign in

REVIEW 3 major objections 3 minor 66 references

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Stitching short guided diffusion rollouts turns OPE's exponential horizon penalty into a small-window penalty.

desk verdict A genuinely new OPE method with strong empirical results, but the central theorem is proven for the learned reward model, not the true return, so the exponential-improvement claim needs a fix. read the letter →

arxiv 2505.20781 v1 pith:H3WZO5NX submitted 2025-05-27 cs.RO cs.LG

Hossein Goli , Michael Gimelfarb , Nathan Samuel de Lara , Haruki Nishimura , Masha Itkina , Florian Shkurti This is my paper · ORCID
classification cs.ROcs.LG
keywords off-policyevaluationdiffusionmodelstrajectorystitchingguidedimportancesamplingmodel-basedreinforcementlearninglong-horizoncontrolpolicyguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STITCH-OPE claims that off-policy evaluation in long-horizon, high-dimensional control can avoid the exponential variance of importance sampling by using a conditional diffusion model trained on short behavior sub-trajectories and stitching the generated chunks together end-to-end. During denoising, the sampler is guided by the score of the target policy minus the score of the behavior policy, which is the gradient of the importance-weight log-likelihood ratio. The paper proves an MSE bound (Theorem 3.3) in which the model-fit error $\delta_\beta$ is multiplied by $\kappa^w$ rather than $\kappa^T$, and the horizon factor appears only as $(T/w)^2$; since $w$ is a small fixed window, this is an exponential reduction in error relative to full-trajectory importance weighting. If the claim holds, a single pretrained behavior diffusion model can evaluate many target policies without retraining, including diffusion policies whose densities are intractable.

What carries the argument

The load-bearing object is a conditional denoising diffusion model $\epsilon_\theta(\tau^k_{t:t+w}, k|s^0_t)$ trained to denoise length-$w$ sub-trajectories given the first state of the chunk. The factorization $p^w_\pi(\tau)=\prod_{t}p_\beta(\tau_{wt:w(t+1)}|s^0_{wt})\prod_{u}\pi(a_u|s_u)/\beta(a_u|s_u)$ lets the target distribution be reached by guiding each chunk's backward diffusion with $g=\alpha\nabla_\tau\sum\log\pi(a|s)-\lambda\nabla_\tau\sum\log\beta(a|s)$, so the importance weight acts as the classifier in classifier guidance. The proof then combines Assumption 3.1 (bounded density ratio $\pi/\beta\le\kappa$) with Assumption 3.2 (total variation between learned and true behavior chunk distributions at most $\delta_\beta$) to turn per-chunk model error into the $\kappa^w\delta_\beta$ terms in the MSE bound. The theorem's force is that $\kappa$ and $\delta_\beta$ are raised to the window length $w$, not the horizon $T$, and that the $(T/w)^2$ factor is only quadratic.

What would settle it

On a small MDP where the true dynamics and both policies are known, generate trajectories from the implemented STITCH-OPE sampler and from the exact reweighted distribution $p_\beta(\tau)\prod_t\pi(a_t|s_t)/\beta(a_t|s_t)$, then estimate their total variation distance; if the TV error grows with guidance strength $\alpha$ or is not controlled by $\kappa^w\delta_\beta$, the theorem does not describe the algorithm that was run.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that trajectory-level importance weighting can be moved inside the denoising process. STITCH-OPE writes the target trajectory distribution as a product of behavior sub-trajectory distributions times the windowed likelihood ratio $\prod_{u=wt}^{w(t+1)-1}\pi(a_u|s_u)/\beta(a_u|s_u)$, and samples from this reweighted distribution by shifting the diffusion mean by the guidance function $\alpha\nabla_\tau\sum\log\pi(a|s)-\lambda\nabla_\tau\sum\log\beta(a|s)$. The paper's main result, Theorem 3.3, bounds the MSE of the return estimator by $(2B_w\kappa^w\delta_\beta/(1-\gamma^w))^2 + 10(T/w)^2B_w^2\kappa^w\delta_\beta + 8B_w^2\kappa^w\delta_\beta/(1-\gamma^{2w}) + \mathrm{Var}_{p_\pi}(J)$, with $B_w$ the maximum per-chunk discounted return. Because the bound depends on the window size $w$ instead of the full horizon $T$, the paper concludes that STITCH-OPE achieves an exponential reduction in MSE versus both importance sampling and length-$T$ trajectory diffusion. On D4RL and OpenAI Gym benchmarks, the method reports better LogRMSE, Spearman correlation, and Regret@1 than FQE, DR, IS, DRE, model-based rollouts, and Policy-Guided Diffusion in most settings.

Load-bearing premise

The proof assumes the guided denoiser samples exactly from the reweighted distribution $p_\beta(\tau)\prod_t\pi(a_t|s_t)/\beta(a_t|s_t)$, but the implemented algorithm uses first-order classifier guidance, which is only an approximation of that reweighted distribution, and the paper gives no bound on that approximation error.

Editorial extensions

If this is right

  • If Theorem 3.3 holds for the implemented sampler, the estimator's dominant error scales like $\kappa^w\delta_\beta$, so choosing a moderate window $w$ (the paper uses $w=8$ on D4RL) turns an exponential-in-horizon penalty into a much smaller penalty.
  • A single diffusion model pretrained on behavior data can evaluate any number of target policies by swapping the score functions at inference time, removing the need to retrain per policy.
  • Because evaluation only needs $\nabla\log\pi$ rather than the full density $\pi(a|s)$, diffusion policies can be evaluated directly, which importance sampling and doubly robust estimators cannot do.
  • The rank-correlation and Regret@1 results mean the estimator can support offline policy selection, not just value prediction.
  • The estimator's variance cannot go below $\mathrm{Var}_{p_\pi}(J)$, the intrinsic variance of the environment and target policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Theorem 3.3 is proven for an exact reweighted sampler; until a Taylor-error bound for the first-order guidance of Eq. (4) is supplied, the exponential improvement claim should be read as conditional on that approximation being accurate.
  • The tempered guidance choice $0<\lambda<\alpha$ amounts to sampling from a tilted posterior over trajectories; sweeping $\lambda$ could produce calibrated uncertainty sets for the value estimate, though the paper does not propose this.
  • The stitching mechanism is not inherently OPE-specific, so the same conditioned sliding-window diffusion could transfer to model-based planning or offline policy optimization, where compositionality and compounding errors are the same bottlenecks; that transfer is untested.
  • The entropy argument predicts stitching helps most when behavior data is multi-modal, so a direct test would build a two-mode MDP and compare STITCH-OPE's mismatch against full-horizon diffusion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. STITCH-OPE proposes a model-based, diffusion-based method for off-policy evaluation (OPE) in high-dimensional, long-horizon continuous-control tasks. The method trains a conditional diffusion model on short behavior-policy sub-trajectories, then generates target-policy trajectories by stitching such sub-trajectories while guiding the denoising process with the score of the target policy minus the score of the behavior policy. The paper's main theoretical claim is Theorem 3.3: under a bounded importance ratio and a bounded total-variation error of the learned behavior model, the MSE of the STITCH-OPE return estimator is bounded by terms that depend on the window size w rather than the full horizon T, yielding an alleged exponential improvement over importance sampling and full-trajectory diffusion. The empirical evaluation compares STITCH-OPE with FQE, DR, IS, DRE, MB, and PGD on D4RL and OpenAI Gym benchmarks, reporting improvements in log-RMSE, rank correlation, and Regret@1, plus ablations over the guidance coefficients and window size.

Significance. The two algorithmic ideas — trajectory stitching with conditional diffusion and negative behavior-policy guidance — are natural and potentially useful for OPE, and the paper's empirical study is more thorough than is typical for a first diffusion-based OPE paper: it includes multiple benchmarks, multiple target-policy classes, five seeds, normalized metrics, and systematic ablations over α, λ, and w. The toy GaussianWorld experiments also help isolate the compositional and distribution-shift effects. However, the central theoretical guarantee is not established as stated: the proof of Theorem 3.3 bounds error relative to a learned reward model, not the true return, and the implemented guided sampler is not the exact reweighted distribution assumed in Lemma D.5. Because the headline contribution is the exponential-in-w MSE bound, these gaps are load-bearing. The empirical results are promising, but the paper currently overclaims its theoretical support; a careful revision that either proves the bound for the actual estimator or substantially softens the theoretical claims is needed.

major comments (3)
  1. [§3.4, Theorem 3.3; Appendix D.1, Eq. (D.1)] The statement of Theorem 3.3 defines J(π) as the expected return under the true trajectory distribution p_π and B_w using the true reward R, but the proof in Appendix D.1 defines J(π) := E_{p_π}[\hat J] with Y_i = Σ_{j=0}^{w-1} γ^j \hat R(s_{iw+j}, a_{iw+j}) in (D.1), and every subsequent bias and variance bound is derived relative to this learned-reward return. The proof never bounds |E_{p_π}[Σ_t γ^t(\hat R(s_t,a_t) − R(s_t,a_t))]|. Even under δ_β = 0 and exact guided sampling, the right-hand side of Theorem 3.3 reduces to Var_{p_π}(J), while the estimator's bias against the true return can be as large as sup_{s,a}|\hat R−R|/(1−γ), which is unaccounted for. The central claim of exponential MSE reduction versus importance sampling is therefore not supported for the quantity that the theorem states.
  2. [Lemma D.5; Algorithm 2, lines 7–10] Lemma D.5 writes the guided model distribution as \hat p_β^{(w)} times the exact importance ratio, and Theorem 3.3 inherits this exact-reweighting assumption through δ_π ≤ κ^w δ_β. The implemented sampler in Algorithm 2 instead uses first-order classifier guidance (Eq. (4)) with per-step normalized gradients and tempered coefficients α and λ (Eq. (6), Section 3.2). No total-variation bound, or any other error control, is provided for the first-order Taylor approximation, the per-step gradient normalization, or the tempering distortion. Consequently, the total-variation bound δ_π ≤ κ^w δ_β is not established for the algorithm that is actually evaluated in the experiments.
  3. [Appendix D.2, Lemma D.7 and Theorem D.8] Theorem D.8 bounds |E_{\hat p_π}[Y_i] − E_{p_π}[Y_i]| using the total variation of unconditional chunk marginals, but Lemma D.5 bounds only conditional TV given a fixed boundary state, and Lemma D.7 assumes both models share the same boundary-state marginal µ(s). In STITCH-OPE, boundary states are produced by previously generated chunks, so the boundary-state distributions under p_π and \hat p_π differ. The proof supplies no bound on this cumulative boundary-state distribution shift, so the unconditional TV bound used in Theorem D.8 does not follow from Assumptions 3.1 and 3.2 as written.
minor comments (3)
  1. [Appendix D.1] The chunk definition S_i := (s_{iw}, a_{iw}, …, s_{(i+1)w}) includes the next boundary state, so X_{i+1} is a deterministic component of S_i and the statement p(X_{i+1}|S_i) = p(X_{i+1}|X_i) is degenerate; this makes the graphical model in Figure 4 ambiguous and should be clarified.
  2. [Section 1 and abstract] The abstract and the remark after Theorem 3.3 promise an exponential reduction in variance, but Theorem 3.3 bounds mean squared error, which also contains a squared bias term; the terminology should be aligned with what is actually proved.
  3. [Section 4.3, Table 3] The text says STITCH-OPE outperforms all other baselines in 6 out of 9 instances, but IS and DR are omitted from the diffusion-policy comparison because they require explicit action densities; the comparison set should be stated explicitly in the text so the claim is not misread as covering all baselines from Table 2.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the theoretical derivation is a self-contained TV/bias-variance argument, though two unproved identifications (reward-model target and exact guidance) are correctness gaps rather than circular reductions, and benchmark hyperparameter tuning is a mild leakage.

full rationale

I walked the derivation chain and found no load-bearing step that reduces to its own inputs. Theorem 3.3 follows from Assumptions 3.1 and 3.2, the chunk decomposition in Appendix D.1, Lemma D.5's TV bound, and the bias/variance decompositions in Theorems D.8 and D.11; these are standard inequalities, not self-citations or fitted parameters renamed as predictions. The guidance function uses policy scores for pi and beta supplied externally, and the diffusion model is trained only on behavior data, so the target-policy evaluation is not defined in terms of the estimator's output. The main caveats are correctness gaps that I do not classify as circularity: (i) Appendix D.1 defines J(pi) := E_{p_pi}[sum_i gamma^{i w} Y_i] with the learned reward model R-hat, whereas the main-text Theorem 3.3 states that J is the true return with B_w built from the true R(s,a); no bound on the reward-model bias |E_{p_pi}[sum_t gamma^t (R-hat - R)]| appears in Theorem D.12, so the exponential-MSE claim against the true return is not established by the proof as written. (ii) Lemma D.5 assumes p-hat_pi^(w) = p-hat_beta^(w) * prod pi/beta, but Algorithm 2 implements first-order classifier guidance (Eq. 4) with per-step normalization, and no TV bound is given for this approximation. These are unproven identifications, not reductions by construction. The mild circular content is experimental: alpha, lambda, and w are selected using the same D4RL/Gym benchmarks whose ground-truth returns appear in Tables 2-3 (see Appendices J and L.1), which is a common but real leakage in OPE model selection. This does not infect the theoretical derivation, so the overall circularity score is low.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new entities are postulated. The central claim rests on the bounded-ratio and model-fit assumptions, plus two unstated idealizations: exact classifier guidance and exclusion of reward-model error. The guidance coefficients and window size are fitted to validation performance.

free parameters (3)
  • alpha (target policy guidance coefficient) = 0.5 (D4RL), 0.1 (Gym)
    Selected based on validation performance; ablations show optimum around alpha in [0.1, 0.5].
  • lambda (behavior policy guidance coefficient) = 0.25 (D4RL), 0.1 (Gym)
    Set as a fraction of alpha (lambda/alpha = 0.5 for D4RL, 1 for Gym); tuned empirically.
  • w (sub-trajectory window size) = 8 (D4RL), 16 (Gym), 4 (HalfCheetah)
    Chosen per domain based on ablations; best performance at moderate w.
assumptions (5)
  • domain assumption Bounded likelihood ratio pi(a|s)/beta(a|s) <= kappa for all s,a.
    Assumption 3.1, standard in OPE; required for the kappa^w terms in Theorem 3.3.
  • domain assumption Total variation between the true and learned chunk-wise behavior distributions is bounded by delta_beta.
    Assumption 3.2; assumes the diffusion model fits the behavior sub-trajectory distribution well.
  • ad hoc to paper First-order Taylor approximation of classifier guidance is exact or its error is negligible.
    Eq (2) and Eq (4) use a first-order Taylor expansion of log p(y|x); no error bound is included in Theorem 3.3.
  • ad hoc to paper The learned reward model R_hat defines the target J(pi) in the proof, so reward model bias is excluded from the bound.
    In Appendix D.1, J(pi) := E_{p_pi}[sum gamma^{iw} Y_i] with Y_i built from R_hat, but the main text presents J(pi) as the true return.
  • ad hoc to paper Tempered guidance with alpha != lambda inherits the same theoretical bounds as alpha = lambda = 1.
    Theorem 3.3 is stated for the exact likelihood-ratio reweighting, but experiments use alpha=0.5, lambda=0.25 for D4RL; no extension is proven for the tempered posterior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation." pith.science (2026). https://pith.science/paper/H3WZO5NX

@misc{pith2026250520781,
  author       = {Pith},
  title        = {Pith review of: STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3WZO5NX}},
  note         = {Machine review of arXiv:2505.20781}
}
read the original abstract

Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial in domains such as robotics or healthcare where direct interaction with the environment is costly or unsafe. Existing OPE methods are ineffective for high-dimensional, long-horizon problems, due to exponential blow-ups in variance from importance weighting or compounding errors from learned dynamics models. To address these challenges, we propose STITCH-OPE, a model-based generative framework that leverages denoising diffusion for long-horizon OPE in high-dimensional state and action spaces. Starting with a diffusion model pre-trained on the behavior data, STITCH-OPE generates synthetic trajectories from the target policy by guiding the denoising process using the score function of the target policy. STITCH-OPE proposes two technical innovations that make it advantageous for OPE: (1) prevents over-regularization by subtracting the score of the behavior policy during guidance, and (2) generates long-horizon trajectories by stitching partial trajectories together end-to-end. We provide a theoretical guarantee that under mild assumptions, these modifications result in an exponential reduction in variance versus long-horizon trajectory diffusion. Experiments on the D4RL and OpenAI Gym benchmarks show substantial improvement in mean squared error, correlation, and regret metrics compared to state-of-the-art OPE methods.

Figures

Figures reproduced from arXiv: 2505.20781 by the authors.

Figure 1
Figure 1. A conceptual illustration of STITCH-OPE, with novel contributions highlighted in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Mean overall performance of all baselines, averaged across environments. Error bars [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Pedagogical example illustrating guided diffusion sample generation for a Gaussian [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Illustration of the sub-trajectory decomposition. Each chunk [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Smoothed performance landscape for Hopper. [PITH_FULL_IMAGE:figures/full_fig_p031_5.png]
Figure 6
Figure 6. Figure 6: Smoothed performance landscape for Walker2d. Results are generally consistent with [PITH_FULL_IMAGE:figures/full_fig_p031_6.png]
Figure 7
Figure 7. Figure 7: Sensitivity of STITCH-OPE to window size [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity of STITCH-OPE to window size [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 9
Figure 9. Figure 9: Trajectory visualizations in the Hopper environment. Both STITCH-OPE and PGD track [PITH_FULL_IMAGE:figures/full_fig_p033_9.png]
Figure 10
Figure 10. Figure 10: Trajectory visualizations in the HalfCheetah environment. STITCH-OPE and PGD both [PITH_FULL_IMAGE:figures/full_fig_p034_10.png]
Figure 11
Figure 11. Figure 11: Trajectory visualizations in the Walker2d environment. STITCH-OPE effectively imitates [PITH_FULL_IMAGE:figures/full_fig_p035_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 51 canonical work pages

  1. [1]

    A. Ajay, Y . Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision making? InThe Eleventh International Con- ference on Learning Representations, 2023. URL https://openreview.net/forum?id= sP1fo2K9DFG

    work page 2023

  2. [2]

    Alquier and J

    P. Alquier and J. Ridgway. Concentration of tempered posteriors and of their variational approximations.The Annals of Statistics, 48(3):1475–1497, 2020

    work page 2020

  3. [3]

    C. M. Bishop and N. M. Nasrabadi.Pattern recognition and machine learning, volume 4. Springer, 2006

    work page 2006

  4. [4]

    Brandfonbrener, W

    D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna. Offline rl without off-policy evaluation.Advances in neural information processing systems, 34:4933–4946, 2021

    work page 2021

  5. [5]

    Brockman

    G. Brockman. Openai gym.arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  6. [6]

    Adaptive reduced tempering For Bayesian inverse problems and rare event simulation

    F. Cérou, P. Héas, and M. Rousset. Adaptive reduced tempering for bayesian inverse problems and rare event simulation.arXiv preprint arXiv:2410.18833, 2024

    work page Pith review arXiv 2024

  7. [7]

    B. Chen, D. M. Monsó, Y . Du, M. Simchowitz, R. Tedrake, and V . Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  8. [8]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

Show all 66 references
  1. [9]

    Dankwa and W

    S. Dankwa and W. Zheng. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent. InProceedings of the 3rd international conference on vision, image and signal processing, pages 1–5, 2019

    2019

  2. [10]

    Dhariwal and A

    P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  3. [12]

    Dudık, D

    M. Dudık, D. Erhan, J. Langford, and L. Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485–511, 2014

    2014

  4. [13]

    Farajtabar, Y

    M. Farajtabar, Y . Chow, and M. Ghavamzadeh. More robust doubly robust off-policy evaluation. InInternational Conference on Machine Learning, pages 1447–1456. PMLR, 2018

    2018

  5. [14]

    Fard and J

    M. Fard and J. Pineau. Pac-bayesian model selection for reinforcement learning.Advances in Neural Information Processing Systems, 23, 2010

    2010

  6. [15]

    J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  7. [16]

    J. Fu, M. Norouzi, O. Nachum, G. Tucker, ziyu wang, A. Novikov, M. Yang, M. R. Zhang, Y . Chen, A. Kumar, C. Paduraru, S. Levine, and T. Paine. Benchmarks for deep off-policy evaluation. InInternational Conference on Learning Representations, 2021. URL https: //openreview.net/...

    2021

  8. [17]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [18]

    M. T. Jackson, M. Matthews, C. Lu, B. Ellis, S. Whiteson, and J. N. Foerster. Policy-guided diffusion. InReinforcement Learning Conference, 2024

    2024

  10. [19]

    Janner, Q

    M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1273–1286. Curran Associa...

    2021

  11. [20]

    Janner, Y

    M. Janner, Y . Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. InInternational Conference on Machine Learning, pages 9902–9915. PMLR, 2022

    2022

  12. [21]

    Jiang and L

    N. Jiang and L. Li. Doubly robust off-policy value evaluation for reinforcement learning. In International conference on machine learning, pages 652–661. PMLR, 2016

    2016

  13. [22]

    Jiang, A

    N. Jiang, A. Kulesza, S. Singh, and R. Lewis. The dependence of effective planning horizon on model accuracy. InProceedings of the 2015 international conference on autonomous agents and multiagent systems, pages 1181–1189, 2015

    2015

  14. [23]

    Kalashnikov, A

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V . Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. InConference on robot learning, pages 651–673. PMLR, 2018

    2018

  15. [24]

    Kidambi, A

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning.Advances in neural information processing systems, 33:21810–21823, 2020

    2020

  16. [25]

    Kostrikov and O

    I. Kostrikov and O. Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation.arXiv preprint arXiv:2007.13609, 2020

    2007 arXiv

  17. [26]

    H. Le, C. V oloshin, and Y . Yue. Batch policy learning under constraints. InInternational Conference on Machine Learning, pages 3703–3712. PMLR, 2019

    2019

  18. [27]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  19. [28]

    G. Li, Y . Shan, Z. Zhu, T. Long, and W. Zhang. Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching. InInternational Conference on Machine Learning, pages 28597–28609. PMLR, 2024

    2024

  20. [29]

    G. Li, L. Shi, Y . Chen, Y . Chi, and Y . Wei. Settling the sample complexity of model-based offline reinforcement learning.The Annals of Statistics, 52(1):233–260, 2024

    2024

  21. [30]

    Q. Liu, L. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation.Advances in neural information processing systems, 31, 2018. 11

    2018

  22. [31]

    V . Liu, P. Nagarajan, A. Patterson, and M. White. When is offline policy selection sample efficient for reinforcement learning?arXiv preprint arXiv:2312.02355, 2023

    2023

  23. [32]

    Y . Liu, P. L. Bacon, and E. Brunskill. Understanding the curse of horizon in off-policy evaluation via conditional importance sampling. InInternational Conference on Machine Learning, pages 6184–6193. PMLR, 2020

    2020

  24. [33]

    Y . Liu, W. Chen, Y . Bai, G. Li, W. Gao, and L. Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai.CoRR, 2024

    2024

  25. [34]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. InInternational Con- ference on Learning Representations, 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7

    2019

  26. [35]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. InInterna- tional Conference on Learning Representations, 2022

    2022

  27. [36]

    C. Lu, P. J. Ball, Y . W. Teh, and J. Parker-Holder. Synthetic experience replay. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview. net/forum?id=6jNQ1AY1Uf

    2023

  28. [37]

    L. Mao, H. Xu, X. Zhan, W. Zhang, and A. Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id= EIl9qmMmvy

    2024

  29. [38]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning.nature, 518(7540):529–533, 2015

    2015

  30. [39]

    Mousavi, L

    A. Mousavi, L. Li, Q. Liu, and D. Zhou. Black-box off-policy estimation for infinite-horizon reinforcement learning. InInternational Conference on Learning Representations, 2020

    2020

  31. [40]

    S. A. Murphy, M. J. van der Laan, J. M. Robins, and C. P. P. R. Group. Marginal mean models for dynamic regimes.Journal of the American Statistical Association, 96(456):1410–1423,

  32. [41]

    Nachum, Y

    O. Nachum, Y . Chow, B. Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections.Advances in neural information processing systems, 32, 2019

    2019

  33. [42]

    X. Nie, E. Brunskill, and S. Wager. Learning when-to-treat policies.Journal of the American Statistical Association, 116(533):392–409, 2021

    2021

  34. [43]

    Precup, R

    D. Precup, R. S. Sutton, and S. P. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759–766, 2000

    2000

  35. [44]

    M. L. Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  36. [45]

    Raghu, O

    A. Raghu, O. Gottesman, Y . Liu, M. Komorowski, A. Faisal, F. Doshi-Velez, and E. Brunskill. Behaviour policy estimation in off-policy policy evaluation: Calibration matters.International Conference on Machine Learning: workshop on Causal Machine Learning, 2018

    2018

  37. [46]

    Rigter, J

    M. Rigter, J. Yamada, and I. Posner. World models via policy-guided trajectory diffu- sion.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=9CcgO0LhKG

    2024

  38. [47]

    Ronneberger, P

    O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, page...

    2015

  39. [48]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  40. [49]

    Spearman

    C. Spearman. The proof and measurement of association between two things.The American Journal of Psychology, 15(1):72–101, 1904

    1904

  41. [50]

    Theocharous, P

    G. Theocharous, P. S. Thomas, and M. Ghavamzadeh. Personalized ad recommendation systems for life-time value optimization with guarantees. InInternational Joint Conference on Artificial Intelligence, 2015. URLhttps://api.semanticscholar.org/CorpusID:8081523

    2015

  42. [51]

    Thomas and E

    P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. InInternational Conference on Machine Learning, pages 2139–2148. PMLR, 2016

    2016

  43. [52]

    Uehara and W

    M. Uehara and W. Sun. Pessimistic model-based offline rl: Pac bounds and posterior sampling under partial coverage.CoRR, 2021

    2021

  44. [53]

    Uehara, C

    M. Uehara, C. Shi, and N. Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355, 2022

    2022 arXiv

  45. [54]

    V oloshin, H

    C. V oloshin, H. M. Le, N. Jiang, and Y . Yue. Empirical study of off-policy policy evaluation for reinforcement learning. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021

    2021

  46. [55]

    Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InThe Eleventh International Conference on Learning Representations,

  47. [56]

    T. Xie, Y . Ma, and Y .-X. Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling.Advances in neural information processing systems, 32, 2019

    2019

  48. [57]

    M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian.Advances in Neural Information Processing Systems, 33:6551–6561, 2020

    2020

  49. [58]

    M. Yang, B. Dai, O. Nachum, G. Tucker, and D. Schuurmans. Offline policy selection under uncertainty. InInternational Conference on Artificial Intelligence and Statistics, pages 4376–

  50. [59]

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y . Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization.Advances in Neural Information Processing Systems, 33: 14129–14142, 2020

    2020

  51. [60]

    M. R. Zhang, T. Paine, O. Nachum, C. Paduraru, G. Tucker, ziyu wang, and M. Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=kmqjgSNXby

    2021

  52. [61]

    Zhang, B

    R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. InInternational Conference on Learning Representations, 2020

    2020

  53. [62]

    Zheng, J

    Y . Zheng, J. Li, D. Yu, Y . Yang, S. E. Li, X. Zhan, and J. Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=j5JvZCaDM0

    2024

  54. [63]

    Z. Zhu, H. Zhao, H. He, Y . Zhong, S. Zhang, H. Guo, T. Chen, and W. Zhang. Diffusion models for reinforcement learning: A survey.arXiv preprint arXiv:2311.01223, 2023. 13 STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation Supplementary Material A...

    2023 arXiv

  55. [67]

    curse of horizon

    resulted in more consistent performance, and allowed the guidance coefficients α and λ to be more easily tuned. F Domains We include experiments on the medium datasets from the D4RL offline suite [15], and Pendulum and Acrobot domains from the OpenAI Gym suite [5]. We set the ...

  56. [2001]

    URLhttp://www.jstor.org/stable/3085909

    ISSN 01621459. URLhttp://www.jstor.org/stable/3085909

  57. [2023]

    URLhttps://openreview.net/forum?id=AHvFDPi-FA

  58. [2024]

    URLhttps://arxiv.org/abs/2406.09509. 10

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.