REVIEW 4 major objections 4 minor 42 references
SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that reweighting a tabular world model's sample distribution with DualDICE corrections can match or beat offline Q-learning and uniform Dyna-Q using fewer pre-collected episodes and fewer planning steps.
desk verdict SimuDICE is a clean, honest integration of DualDICE weights into Dyna-style offline planning; the Taxi gains look real but the claimed DICE-confidence balance is not established, and the baseline set is too thin for the abstract's claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sampling distribution $P(s,a)$ over the tabular world model's stored transitions, defined by Eq. (14) as the normalized version of $L(s,a) = C(s,a) + \lambda^{-1} e^{\lambda w^{\pi/D}(s,a)} / \sum_{s',a'} e^{\lambda w^{\pi/D}(s',a')}$. $C(s,a)$ is the empirical confidence (normalized occurrence count) of the transition, and $w^{\pi/D}$ is the DualDICE density ratio $d^\pi(s,a)/d^D(s,a)$. The mechanism works by making Q-learning updates in the planner visit state-action pairs in an order that balances the model's reliability with the policy's distributional shift.
What would settle it
Take a discrete environment where the true transitions are multimodal, such that a tabular model's most-frequent-next-state approximation hides uncertainty, and compare the ordering of state-action pairs induced by $w^{\pi/D}$ computed on the true model versus on the learned tabular model. If the rankings differ materially, SimuDICE's sampling guidance is based on a correction that does not match the model it is drawing from, and the reported sample-efficiency gains would not transfer.
Extended reading notes
Core claim
The central claim is that the sampling probabilities used by a Dyna-style planner can be set to $P(s,a) \propto L(s,a)$, where $L(s,a) = C(s,a) + \lambda^{-1} e^{\lambda w^{\pi/D}(s,a)} / \sum_{s',a'} e^{\lambda w^{\pi/D}(s',a')}$, with $C(s,a)$ the normalized frequency of the state-action pair in the dataset (the model's confidence) and $w^{\pi/D}(s,a)$ the DualDICE stationary distribution correction $d^\pi(s,a)/d^D(s,a)$. With this reweighting, the planner preferentially revisits transitions that are both trusted by the model and favored by the current target policy, and it backs off toward confident transitions when the correction is unreliable. The paper shows empirically that this yields average per-step rewards comparable to or higher than uniform sampling and offline Q-learning while using less data and fewer planning steps, with the gains largest in Taxi, the largest of the three tested environments.
Load-bearing premise
Everything rests on the premise that the DualDICE stationary-distribution corrections, computed against the true environment and the current target policy, still point at the right experiences to reweight when the planner only samples from a simple tabular model that stores the most common next state and average reward for each pair.
Editorial extensions
If this is right
- Matching baseline performance with fewer pre-collected episodes means offline practitioners can get usable policies from smaller datasets in tabular domains.
- Fewer planning steps per iteration reduce the number of synthetic transitions generated, which lowers the chance that an imperfect model produces hallucinated states.
- The formula's robustness across epsilon-greedy data-collection policies suggests that DICE-guided reweighting can adapt when the behavioral policy is noisy.
- Because the ablation found no benefit from multiple reweighting iterations, the method can be run as a single reweighting pass, which is computationally cheaper.
Reading between the lines
- The reweighting formula only reorders observed transitions; it cannot create genuinely new states, so a generative world model paired with the same correction would likely need a stronger confidence safeguard than the empirical frequency $C(s,a)$ used here.
- DualDICE weights are computed against the fixed dataset at each iteration, but the target policy changes during planning; an alternating update that recomputes corrections after each planning round could matter in environments with stronger distribution shift than the three tested.
- A testable extension is to make the regularization parameter $\lambda$ adaptive to the discrepancy between $C$ and $w^{\pi/D}$, rather than fixing it at 1000 as in the reported experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SimuDICE, an offline model-based RL method for discrete tabular domains. It learns a tabular world model from an offline dataset, estimates the stationary distribution correction w^{π/D} between the target policy and the behavior policy using DualDICE, and uses it to reweight state-action pairs sampled from the world model during planning. The reweighting combines a model-confidence term C(s,a) (normalized dataset frequency) with a λ-regularized softmax of the DICE weights. The policy is updated by Q-learning on the synthetic transitions, and the procedure can be iterated. Experiments on Taxi, CliffWalking, and FrozenLake compare SimuDICE against offline Q-learning and a uniform-sampling Dyna-Q variant, with ablations over planning steps, sampling formulas, and the number of DICE-update iterations. The paper claims improved sample efficiency and fewer required planning steps relative to the tested baselines, with the gains concentrated in Taxi.
Significance. If validated, SimuDICE would provide a simple, interpretable mechanism for combining DICE-based distribution correction with model-based planning in tabular offline RL, and the open-source code and clear algorithmic structure are valuable for reproducibility. The study is honest about its limitations, and the ablations over planning steps and sampling formulas are useful. However, the contribution is currently supported by a narrow experimental base—only one environment shows a clear improvement—and there is a load-bearing inconsistency in the definition of the sampling likelihood that makes the claimed mechanism ambiguous. As it stands, the paper establishes a plausible sampling-priority heuristic rather than a demonstrated general method.
major comments (4)
- [§4, Eq. (13); §5.2, Eq. (15)] The likelihood function is defined inconsistently. Eq. (13) as typeset reads L(s,a)=C(s,a)+ e^{w(s,a)λ}/(Σ e^{w(s',a')λ}/λ), i.e., C plus λ times a softmax term, whereas Eq. (15), which is called the default formula, reads L_1(s,a)=C(s,a)+ e^{w(s,a)λ}/(λ Σ e^{w(s',a')λ}), i.e., C plus softmax/λ. These differ by a factor of λ² and have opposite behavior for the reported λ=1000: Eq. (13) makes the DICE term dominate and concentrate nearly all sampling mass on the largest-w pair, while Eq. (15) makes the DICE term contribute only 1/λ of the total likelihood mass, so P(s,a) is essentially C(s,a) after normalization. The statement in §4 that λ 'prevent[s] mode collapse' is compatible with neither interpretation: under Eq. (13) a large λ sharpens the softmax and induces mode collapse, and under Eq. (15) the DICE contribution is negligible. This inconsistency is load-bearing because the paper attributes the Taxi improvements to the interaction of C and w, and the reported hyperparameter λ=1000 must be reconciled with the actual default formula before the experimental results can be interpreted.
- [§5.1, Fig. 3; §6] The central sample-efficiency and 'comparable to existing algorithms' claims are not supported by the presented comparisons. In CliffWalking the paper itself reports 'no significant difference' (Sec. 5.1), and in FrozenLake the only clear gap is that offline Dyna-Q underperforms, with SimuDICE matching offline Q-learning; the consistent advantage appears only in Taxi. No significance tests accompany the word 'significantly.' The two baselines are offline Q-learning and offline Dyna-Q, which is a uniform-sampling variant of SimuDICE itself; no comparison is made to standard offline or offline MBRL algorithms such as CQL, IQL, MOPO, or MOReL. The Abstract's claim of performance 'comparable to existing algorithms' therefore overstates the evidence, and the Section 6 limitations already acknowledge this. This should be fixed either by adding competitive baselines or by explicitly restricting the claim to a comparison against uniform Dyna-style planning.
- [§5; Algorithm 1, line 5] The DualDICE component is not described at implementation level. The paper does not state the function class used for ν, the optimizer, the number of gradient steps, or any clipping or normalization applied to w before it is inserted into Eq. (13)/(15). Additionally, the seed count is inconsistent: Section 5 says results are averaged over '500 plays and 20 seeds,' while the Fig. 3 caption says '5 different random seeds.' These details are necessary to reproduce the reported variance bands and to check whether the estimated w values are stable in the small-sample tabular setting.
- [§4, Algorithm 1; §5.2, Fig. 6] The role of the DICE weights is unclear because in the one-iteration experiments they are computed from the initial target policy and are not updated during planning, and Fig. 6 shows that additional iterations do not change performance. This is consistent with the concern that the reweighting mainly enforces self-consistency with the current policy rather than correcting distribution shift toward a better policy. A concrete test would be to compare SimuDICE against a version that re-estimates w during planning and against a baseline that prioritizes transitions with random weights or with C-only weights; without such a control, the mechanism behind the Taxi gains remains ambiguous.
minor comments (4)
- [§5, first paragraph] The sentence 'The evaluation was performed on the Taxi environment using the 9 datasets collected from the other environments' appears to be a textual error; the results in Fig. 3 are reported for Taxi, FrozenLake, and CliffWalking, each with three datasets.
- [§4, Algorithm 2] The tabular world model returns only the most frequently observed next state, which removes stochasticity from the model; the statement that this approach 'allows the model to handle stochastic environments' is misleading, particularly for FrozenLake where the environment is stochastic.
- [§5.2, Eq. (16)] In Formula 2 the subtractive term e^{wλ}/(λ Σ e^{w'λ}) can exceed C(s,a), making L_2(s,a) negative; after normalization by the total sum this does not define a valid probability distribution. The ablation should either clip negative likelihoods or otherwise ensure nonnegativity.
- [§5.1] The word 'significantly' is used to describe Taxi improvements without any statistical test or reported standard error; the authors should report per-seed results and a paired test or bootstrap confidence intervals.
Circularity Check
No significant circularity; SimuDICE's sampling reweighting is an algorithmic design evaluated against external uniform-sampling baselines.
full rationale
The paper's central claim is empirical: SimuDICE matches or exceeds offline Q-learning and offline Dyna-Q on three tabular environments using fewer data and planning steps. The sampling rule (Eqs. 13-14) is a hand-designed combination of the empirical confidence C and a softmax of the DualDICE weights; these quantities are inputs to the algorithm, not parameters fitted to the evaluation metric. The comparison against offline Dyna-Q, which the paper explicitly identifies as SimuDICE with uniform sampling, provides an external control, and the ablations vary the sampling formula and the number of planning steps/iterations. The iterative dependence of w^{pi/D} on the current policy is a standard algorithmic feature rather than a circular derivation: no equation or theorem in the paper reduces a claimed output to its own input. DualDICE is cited as external prior work, not as a self-citation chain, and the paper's stated limitations (simple environments, sensitivity to the formula, narrow comparison set) are acknowledged weaknesses rather than hidden circularities. The lambda=1000 hyperparameter concern is a robustness or effectiveness question, not evidence that a prediction is forced by construction.
Assumptions & free parameters
free parameters (5)
- lambda (lambda) =
1000
- learning rate (alpha) =
0.1 (0.05 in ablation)
- discount factor (gamma) =
0.99
- planning steps =
10 or 20
- number of iterations =
1 by default
assumptions (5)
- domain assumption The offline dataset D is drawn i.i.d. from an unknown behavioral distribution d_D, with (s,a) ~ d_D, r ~ R, s' ~ T.
- standard math DualDICE optimization converges to the true discounted stationary distribution corrections w^{pi/D} for the policies considered.
- domain assumption A tabular world model that averages rewards and stores the most frequently observed next state is an adequate dynamics model for planning.
- ad hoc to paper C(s,a), normalized dataset frequency, is a valid proxy for world model prediction confidence.
- ad hoc to paper The likelihood formula L(s,a) in Eq. (13)/(15) is correctly specified and yields a normalized probability after Eq. (14).
Cite this review
Pith. "Pith review of SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation." pith.science (2026). https://pith.science/paper/NKAV2HBR
@misc{pith2026241206486,
author = {Pith},
title = {Pith review of: SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKAV2HBR}},
note = {Machine review of arXiv:2412.06486}
}
read the original abstract
In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., Fleuret, F.: Diffusion for world modeling: Visual details matter in atari (2024), https:// arxiv.org/abs/2405.12399
arXiv 2024
-
[2]
Machine Learning20(1-2), 65–81 (1995)
Baird, L.: Residual algorithms: Reinforcement learning with function approxima- tion. Machine Learning20(1-2), 65–81 (1995)
work page 1995
-
[3]
arXiv preprint arXiv:2106.09119 (2021), abs/2106.09119
Cang, C., Rajeswaran, A., Abbeel, P., Laskin, M.: Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl. arXiv preprint arXiv:2106.09119 (2021), abs/2106.09119
arXiv 2021
-
[4]
arXiv preprint arXiv:1809.05214 (2018), https://arxiv.org/abs/1809.05214
Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y., Asfour, T., Abbeel, P.: Model-based reinforcement learning via meta-policy optimization. arXiv preprint arXiv:1809.05214 (2018), https://arxiv.org/abs/1809.05214
arXiv 2018
-
[5]
In: Proceedings of the 28th International Conference on Machine Learning (ICML-11)
Deisenroth, M.P., Rasmussen, C.E.: Pilco: A model-based and data-efficient ap- proach to policy search. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11). pp. 465–472 (2011)
work page 2011
-
[6]
Ding, Z., Zhang, A., Tian, Y., Zheng, Q.: Diffusion world model (2024), https: //arxiv.org/abs/2402.03570
arXiv 2024
-
[7]
arXiv preprint arXiv:2110.02758 (2021)
Eysenbach, B., Khazatsky, A., Levine, S., Salakhutdinov, R.: Mismatched no more: Joint model-policy optimization for model-based rl. arXiv preprint arXiv:2110.02758 (2021)
arXiv 2021
-
[8]
In: Proceedings of the 36th International Conference on Ma- chine Learning
Fujimoto, S., Meger, D., Precup, D.: Off-policy deep reinforcement learning with- out exploration. In: Proceedings of the 36th International Conference on Ma- chine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 2052–2062. PMLR (2019)
work page 2019
Show all 42 references
-
[9]
In: International Conference on Learning Representations (2020)
Hafner,D.,Lillicrap,T.,Norouzi,M.,Ba,J.:Dreamer:Reinforcementlearningwith latent world models. In: International Conference on Learning Representations (2020)
2020
-
[10]
arXiv preprint arXiv:2301.04104 (2023)
Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T.: Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 (2023)
2023 arXiv
-
[11]
arXiv preprint arXiv:2010.02193 (2020)
Hafner, D., Schrittwieser, J., Mankowitz, D., Barreto, A., Lillicrap, T.: Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 (2020)
2020 arXiv
-
[12]
In: Proceedings of the 34th International Conference on Machine Learning (ICML 2017)
Hallak, A., Mannor, S.: Consistent on-line off-policy evaluation. In: Proceedings of the 34th International Conference on Machine Learning (ICML 2017). pp. 1372–
2017
-
[13]
arXiv preprint arXiv:2006.04363 (2020)
Jafferjee, T., Imani, E., Talvitie, E.J., White, M., Bowling, M.: Hallucinating value: A pitfall of dyna-style planning with imperfect environment models. arXiv preprint arXiv:2006.04363 (2020)
2020 arXiv
-
[14]
arXiv preprint arXiv:1906.08253 (2019), https://arxiv.org/ abs/1906.08253
Janner, M., Fu, J., Zhang, M., Levine, S.: When to trust your model: Model-based policy optimization. arXiv preprint arXiv:1906.08253 (2019), https://arxiv.org/ abs/1906.08253
2019 arXiv
-
[15]
arXiv preprint arXiv:1907.00456 (2019), https://arxiv.org/abs/1907.00456
Jaques, N., Ghandeharioun, A., Shen, J.H., Ferguson, C., Lapedriza, À., Jones, N., Gu, S., Picard, R.W.: Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 (2019), https://arxiv.org/abs/1907.00456
2019 arXiv
-
[16]
In: Advances in Neural Information Processing Sys- tems (2020)
Kidambi, R., Rajeswaran, A., Netrapalli, P., Joachims, T.: Morel: Model-based offline reinforcement learning. In: Advances in Neural Information Processing Sys- tems (2020)
2020
-
[17]
arXiv preprint arXiv:2110.06169 (2021) 16 C.E
Kostrikov, I., Nair, A., Levine, S.: Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169 (2021) 16 C.E. Brita et al
2021 arXiv
-
[18]
In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R
Kumar, A., Fu, J., Soh, M., Tucker, G., Levine, S.: Stabilizing off-policy q-learning via bootstrapping error reduction. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R. (eds.) Ad- vances in Neural Information Processing Systems. vol. 32,...
2019
-
[19]
In: Advances in Neural Information Processing Systems
Kumar, A., Fu, J., Tucker, G., Levine, S.: Stabilizing off-policy q-learning via boot- strapping error reduction. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
2019
-
[20]
In: International Conference on Learning Representa- tions (2018), https://arxiv.org/abs/1802.10592
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., Abbeel, P.: Model-ensemble trust- region policy optimization. In: International Conference on Learning Representa- tions (2018), https://arxiv.org/abs/1802.10592
2018 arXiv
-
[21]
arXiv preprint arXiv:2002.04523 (2020)
Lambert, N., Amos, B., Yadan, O., Calandra, R.: Objective mismatch in model- based reinforcement learning. arXiv preprint arXiv:2002.04523 (2020)
2020 arXiv
-
[22]
In: Wiering, M., van Otterlo, M
Lange, S., Gabel, T., Riedmiller, M.: Batch reinforcement learning. In: Wiering, M., van Otterlo, M. (eds.) Reinforcement Learning: State-of-the-Art, chap. 2, pp. 45–73. Springer (2012). https://doi.org/10.1007/978-3-642-27645-3_2
2012 doi
-
[24]
Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: Tutorial, review,andperspectivesonopenproblems.arXivpreprintarXiv:2005.01643(2020)
2020 arXiv
-
[25]
In: Advances in Neural Information Processing Systems 31 (NeurIPS 2018)
Liu, Q., Li, L., Tang, Z., Zhou, D.: Breaking the curse of horizon: Infinite-horizon off-policy estimation. In: Advances in Neural Information Processing Systems 31 (NeurIPS 2018). pp. 5356–5366. Curran Associates, Inc. (2018)
2018
-
[26]
Nature518(7540), 529–533 (2015)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature518(7540), 529–533 (2015)
2015
-
[27]
Journal of the American Statistical Associa- tion 96(456), 1410–1423 (2001)
Murphy, S.A., van der Laan, M.J., Robins, J.M., Group, C.P.P.R.: Marginal mean models for dynamic regimes. Journal of the American Statistical Associa- tion 96(456), 1410–1423 (2001)
2001
-
[28]
In: Advances in Neural Information Processing Systems
Nachum, O., Chow, Y., Dai, B., Li, L.: Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In: Advances in Neural Information Processing Systems. pp. 2315–2325 (2019)
2019
-
[29]
In: Brodley, C.E., Danyluk, A.P
Precup, D., Sutton, R.S., Dasgupta, S.: Off-policy temporal-difference learning with function approximation. In: Brodley, C.E., Danyluk, A.P. (eds.) Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001). pp. 417–424. Morgan Kaufmann, San Franci...
2001
-
[30]
John Wiley & Sons, New York (1994)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Pro- gramming. John Wiley & Sons, New York (1994)
1994
-
[31]
Nature529(7587), 484–489 (2016)
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Master- ing the game of go with deep neural networks and tree search. Nature529(7587), 484–489 (2016). https://doi.org...
2016 doi
-
[32]
Proceedings of the Seventh International Conference on Machine Learning pp
Sutton, R.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning pp. 216–224 (1990)
1990
-
[33]
ACM SIGART Bulletin2(4), 160–163 (1991)
Sutton, R.S.: Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin2(4), 160–163 (1991)
1991
-
[34]
MIT Press, Cambridge, MA, 2 edn
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2 edn. (2018) SimuDICE 17
2018
-
[35]
https://doi.org/10.1016/j.engappai.2021.104366
Swazinna, P., Udluft, S., Runkler, T.A.: Overcoming model bias for robust of- flinedeepreinforcementlearning.EngineeringApplicationsofArtificialIntelligence 104, 104366 (2021). https://doi.org/10.1016/j.engappai.2021.104366
2021
-
[36]
In: Proceedings of the 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS)
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: Proceedings of the 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS). pp. 23–...
2017
-
[37]
https://doi.org/ 10.5281/zenodo.8127026, https://zenodo.org/record/8127025
Towers,M.,Terry,J.K.,Kwiatkowski,A.,Balis,J.U.,Cola,G.d.,Deleu,T.,Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J.J., Shen, A.T.J., Younis, O.G.: Gymnasium (Mar 2023). https://doi.org/ 10.5281/zenodo.8127026, https://zenod...
2023
-
[38]
In: Advances in Neural Information Processing Systems 9 (NIPS 1996)
Tsitsiklis, J.N., Van Roy, B.: Analysis of temporal-difference learning with function approximation. In: Advances in Neural Information Processing Systems 9 (NIPS 1996). pp. 1075–1081. MIT Press (1996)
1996
-
[39]
CoRR abs/1911.11361 (2019)
Wu, Y., Tucker, G., Nachum, O.: Behavior regularized offline reinforcement learn- ing. CoRR abs/1911.11361 (2019)
2019 arXiv
-
[40]
In: International Conference on Learning Representa- tions
Yang, M., Nachum, O., Dai, B., Li, L., Schuurmans, D.: Off-policy evaluation via the regularized lagrangian. In: International Conference on Learning Representa- tions. ICLR (2020)
2020
-
[41]
In: Advances in Neural Information Processing Systems (2020)
Yu,T.,Kumar,A.,Zhang,S.,Gao,G.,Levine,S.:Mopo:Model-basedofflinepolicy optimization. In: Advances in Neural Information Processing Systems (2020)
2020
-
[42]
In: International Conference on Learning Representations (2020)
Zhang, R., Dai, B., Li, L., Schuurmans, D.: Gendice: Generalized offline estima- tion of stationary values. In: International Conference on Learning Representations (2020)
2020
-
[43]
arXiv preprint arXiv:2001.11113 (2020)
Zhang, S., Liu, B., Whiteson, S.: Gradientdice: Rethinking generalized offline esti- mation of stationary values. arXiv preprint arXiv:2001.11113 (2020)
2020 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.