Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a policy-regularization term that keeps the learned policy close to the MPC planner's behavior reduces persistent value overestimation and substantially improves TD-MPC2 on high-dimensional continuous control.

desk verdict A simple policy-constraint patch to TD-MPC2 that gets large empirical gains on high-dimensional control, but the theory is sloppier than the experiments and the HumanoidBench baseline needs a controlled re-run. read the letter →

arxiv 2502.03550 v1 pith:LGMCAM66 submitted 2025-02-05 cs.LG cs.RO

Haotian Lin , Pengcheng Wang , Jeff Schneider , Guanya Shi This is my paper · ORCID
classification cs.LGcs.RO
keywords model-basedreinforcementlearningmodelpredictivecontroltemporaldifferencevalueoverestimationpolicyconstraintout-of-distributionactionshumanoidcontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Temporal-difference model predictive control learns a value function by bootstrapping a nominal policy while collecting data with an MPC planner, and this paper argues that the mismatch between those two policies causes persistent value overestimation that standard training never corrects. The overestimation grows with action dimensionality, reaching hundreds or thousands of percent error on 61-DoF humanoid tasks, and it degrades the planner's terminal-value estimates. The proposed fix is a single regularization term in the policy loss that keeps the learned policy near the actions the planner actually took, reducing out-of-distribution queries to the value function. With that term, value estimates track true returns much more closely, and the algorithm outperforms the TD-MPC2 baseline on high-dimensional continuous control, including an average improvement of over 100% on HumanoidBench. A sympathetic reader would take the paper's central claim to be that policy mismatch, not just function-approximation noise, is the root cause and that a minimal distribution constraint is sufficient to fix it.

What carries the argument

The central object is the policy-regularization term $\beta\log\mu(a|s)$ added to the maximum-entropy policy objective in Eq. 10, where $\mu$ is the behavior policy, a weighted mixture of past planner policies $\pi_{H,k}$. Because exact $\log\mu$ is unavailable, the implementation maximizes $\mathbb{E}_{\mu'\sim\{\mu\}}[\log\mu']$ over the planner actions stored in the replay buffer, treating that as a lower bound. This term is the Lagrangian relaxation of the KL-constrained policy improvement problem in Eq. 8, and it works by penalizing the learned policy for selecting actions the planner would not take, keeping value-function queries inside the support of the data distribution.

What would settle it

Compare TD-M(PC)2 against a version that uses the exact planner log-density $\log\mu(a|s)$ computed from the MPC Gaussian parameters instead of the stored-action surrogate. If performance and value error change materially, the lower-bound surrogate is doing the work; if they do not, the claimed mechanism is not the source of the gains.

Watch

Extended reading notes

Core claim

The paper claims that in the TD-MPC2 pipeline the data-generating policy is the H-step lookahead planner $\pi_H$, while value targets bootstrap the nominal policy $\pi$, and the resulting distributional mismatch makes value errors invisible to the training distribution. Theorems 3.1, 4.1, and 4.2 formalize how approximation error accumulates through policy iteration, widens the performance gap between $\pi_H$ and $\pi$, and forces the two policies to diverge, so the overestimated regions are never revisited and corrected. The remedy is distribution-constrained policy improvement: replace the unconstrained policy update with the KL-constrained problem of Eq. 8 and implement its Lagrangian form as the policy loss $L_\pi = -\mathbb{E}_{a\sim\pi}[Q(s,a) - \alpha\log\pi(a|s) + \beta\log\mu(a|s)]$, using the stored planner actions as a surrogate for the behavior policy $\mu$. Empirically this reduces value approximation error sharply on 4-DoF, 36-DoF, and 61-DoF tasks and yields large performance gains, especially on HumanoidBench locomotion.

Load-bearing premise

The load-bearing premise is that maximizing the log-likelihood of stored planner actions is a faithful lower bound on the true planner policy's density, so the extra term really constrains the learned policy toward in-distribution actions rather than merely imitating the planner.

Editorial extensions

If this is right

  • Value estimates in TD-M(PC)2 track Monte Carlo returns far more closely than those of TD-MPC2, so the planner's terminal bootstrapping becomes more trustworthy in high-dimensional tasks.
  • On the 14-task HumanoidBench locomotion suite, the method is reported to improve over TD-MPC2 by over 100% on average, with the largest gains on Run, Slide, and Pole.
  • On DMControl's seven high-dimensional tasks, the method slightly outperforms the baseline overall, with clear gains on the three dog tasks and comparable performance on humanoid tasks.
  • The modification costs no additional computation and no per-task hyperparameter tuning, so it can be applied to existing TD-MPC-style implementations with a few lines of code.
  • Because the constraint is applied to the policy prior rather than the planner, online exploration is not restricted; conservatism enters only where values are bootstrapped.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mismatch diagnosis generalizes, the same one-term constraint should transfer to other planning-plus-value algorithms, including diffusion-based planners, since the failure mode is structural rather than specific to TD-MPC2.
  • The ablation showing near-parity of a pure behavior-cloning variant suggests that conservatism itself, not improved value learning, may be the active ingredient; a $\beta$-annealing schedule could separate imitation from value-based improvement.
  • A direct confirmation would be to track the total-variation divergence between $\pi_H$ and $\pi$ over training; the paper predicts this divergence should shrink under TD-M(PC)2 while value error falls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper claims that TD-MPC2-style algorithms suffer from persistent value overestimation caused by a structural mismatch between the planner-driven data-collection policy π_H and the learned nominal policy π. It proposes TD-M(PC)2, a minimalist modification that adds a policy regularization term to the actor loss so that the learned policy stays close to the behavior policy, thereby reducing out-of-distribution value queries. The paper supports this claim with theoretical bounds (Theorems 3.1, 4.1, 4.2), empirical demonstrations of overestimation and its mitigation (Figures 1, 5, 7), benchmark results on HumanoidBench and DMControl, and ablation studies. The headline empirical result is that TD-M(PC)2 improves over TD-MPC2 by over 100% on the HumanoidBench locomotion average.

Significance. If the stated mechanism is correct, the paper offers a simple, general, and computationally cheap improvement to a prominent MBRL pipeline, with particularly large gains in very high-dimensional control tasks. The theoretical analysis connecting policy mismatch to error accumulation is a valuable addition to the literature, and the paper is transparent about the looseness of its bounds. However, the link between the theoretical constrained policy-improvement formulation (Eq. 8) and the implemented loss (Eq. 10) is not presently coherent, and the ablation results suggest that the observed gains may be driven more by conservatism than by the claimed value-learning improvement. The empirical comparison against HumanoidBench baselines that were not re-run under the paper's own protocol also weakens the headline quantitative claim. These issues are fixable, but they are central to the paper's contribution as currently stated.

major comments (4)
  1. [Section 5.2, Equations (8), (10), (23)] The claimed relationship between the constrained policy improvement and the implemented loss is not correct as written. The Lagrangian of (8) with an entropy bonus αH(π) is E_{a∼π}[Q(s,a)] − (α+β) E_{a∼π}[log π(a|s)] + β E_{a∼π}[log μ(a|s)], whereas Eq. (10) has coefficient α on the log π term. With β=1.0 and α=1e−4 (Table 1), the missing β log π term is the dominant entropy contribution. Moreover, the sentence 'we can maximize E_{μ′∼{μ}}[log μ′] as the lower bound of log(μ)' is not mathematically meaningful: for a fixed μ, E_{μ′∼μ}[log μ′(a)] is a constant and is not a lower bound for log μ(a|s) at actions sampled from π. Equation (23) is coherent only if μ_t is interpreted as the planner density evaluated at π-sampled actions, but that interpretation conflicts with the stored-action wording in Algorithm 1 and with the behavior-cloning variant in Eq. (25). Because the paper's central claim is that the added term implements a policy constraint that reduces out-of-distribution value queries, the authors must state precisely what loss is implemented (ideally with the actual code), and either derive it from (8) or explicitly present it as a heuristic cross-entropy regularizer.
  2. [Section 6.2, Figure 3] The HumanoidBench comparison relies on author-provided TD-MPC2 scores from an external repository (humanoid-bench) rather than on runs under the paper's own protocol. The headline claim that TD-M(PC)2 'improves TD-MPC2 by over 100%' therefore depends on an uncontrolled baseline; differences in seeds, environment versions, or evaluation procedures could change the comparison. Please either re-run TD-MPC2 under identical conditions for all tasks or clearly restrict the claim to the externally reported numbers and report the baseline's seed count and protocol.
  3. [Figures 1, 5, 7 and Appendix C] The ground-truth value estimates are Monte Carlo averages over 100 episodes and are plotted without confidence intervals or per-seed traces. The claim of 'persistent value overestimation' is the paper's motivation, and the proposed reduction in overestimation is a central result; without error bars it is hard to assess whether the Hopper-Stand case (15% error) or the convergence behavior in Figure 7 is significant. Please add confidence intervals or per-seed curves, and state the number of episodes and seeds used for the true-value estimator.
  4. [Section 6.3, Equation (25)] The 'pure behavior cloning' variant is not behavior cloning in the standard sense. Equation (25) maximizes E_{s∼B} E_{a∼π(·|s)} log μ(a|s), i.e., it minimizes KL(π∥μ), while BC normally maximizes the log-likelihood of stored actions under π, E_{(s,a)∼B} log π(a|s). As written, the ablation shows that a conservative policy update (whether or not it uses the Q-function) is sufficient for most of the gain, but it does not isolate the mechanism of improved value learning through reduced OOD queries. Please correct the description of the variant and, if the mechanism claim is to be retained, add an ablation that holds the policy update fixed and measures value error (e.g., Figure 5) for the full method versus the BC variant.
minor comments (7)
  1. [References, [19]] Reference [19] is not Hafner et al.'s Dreamer paper; it is a database paper for emotion recognition. Please correct the citation for Dreamer.
  2. [Theorem 3.1, Eq. (4)] The phrase 'for any policy µ' after the displayed equation is dangling and does not connect to the theorem statement.
  3. [Figures 1 and 5] The x-axis label 'Training Iterations (x10)' is ambiguous; it should read 10^5 or 1e5 steps.
  4. [Algorithm 1] The line 'Initialize ... by pertaining on uniformly sampled data' contains a typo; it should be 'pretraining'.
  5. [Figure 8 caption] The caption 'Constrained policy update through A W ACMean' is missing a space, and 'ramdom' is a typo for 'random'.
  6. [Appendix B, Eq. (23) and (25)] The text calls this a 'TD3-BC style policy constraint,' but TD3-BC uses a squared-distance penalty on actions, not a log-density term; please clarify the analogy.
  7. [Appendix A.7, Theorem 4.1 proof] In the displayed chain of equalities, the term '+ γ^H E_{ˆτ_k}[V^{π_{k−1}}(s_H) + \hat V_{k−1}(s_H)]' should have a minus sign before the value function to be consistent with the preceding line.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is not forced by its inputs; the Section 5.2 Lagrangian mismatch is a correctness/fidelity issue, not a circular reduction.

full rationale

The paper's central claim is that SAC-style policy iteration with planner-generated data causes persistent value overestimation and that a policy-regularization term mitigates it. This is not circular: the overestimation is measured against Monte Carlo returns (Figure 1), the policy mismatch is defined independently as a KL/performance gap (Theorems 4.1 and 4.2), and the proposed loss (Eq. 10, implemented as Eq. 23) is a modification of a known TD3-BC-style constraint (cited to [6]), not a fitted parameter that is then reported as a prediction. The empirical validation is against external benchmarks (TD-MPC2, DreamerV3, SAC) with no task-specific tuning, so the claimed improvements are not constructed from the method's own outputs. There are no self-citations by the present authors, and Theorem 3.1 is explicitly adopted from LOOP [34], which is independent prior work. The most serious flaw is in Section 5.2: Eq. 10 is not the Lagrangian of Eq. 8 (the beta log pi term is absent), and the sentence 'we can maximize E_{mu'~{mu}}[log mu'] as the lower bound of log(mu)' is mathematically incoherent as written. This is a real implementation-fidelity and correctness concern, and the BC ablation in Figure 6 suggests the gains may come from imitation rather than the stated KL mechanism. However, an unproved or even incorrect surrogate does not make the argument circular: the method is fully specified, the value-learning improvement is tested empirically, and no conclusion is assumed by construction. The approximation weakens the theoretical link between Eq. 8 and the implemented loss, but it does not reduce the paper's claims to their inputs. Under the hard rule that circularity requires exhibiting a specific reduction of a claimed result to a fit or to a self-citation chain, no such step exists here.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard RL assumptions plus two ad-hoc simplifications in the implementation: replacing the behavior policy with stored planner actions and treating a TD3-BC-style objective as a convenient lower bound. These are implementation choices rather than fitted parameters, and the method's performance is robust to the main hyperparameter β.

free parameters (2)
  • Prior constraint coefficient β = 1.0
    Chosen as the default in all experiments; the ablation over β=1.0 and β=0.05 shows insensitivity, so it is not fitted to the target result.
  • Scale threshold S = 2.0
    Controls when the policy constraint is activated via moving percentiles of Q; set by hand following TD-MPC2 practice.
assumptions (4)
  • domain assumption The behavior policy μ can be represented as a weighted sum of past H-step lookahead policies π_H (Section 5.1).
    This representation is used to justify constraining the policy toward the planner's action distribution; in practice the stored planner action is used as a surrogate.
  • ad hoc to paper Stored planner actions a_t are a valid empirical surrogate for the behavior policy μ in the constraint (Section 5.2).
    This substitution is not derived; it equates the behavior policy with the instantaneous planner outputs stored in the buffer.
  • ad hoc to paper Maximizing E_{μ'∼{μ}}[log μ'] is a valid lower bound for log μ(a|s) (Section 5.2).
    Stated without proof; the actual implementation uses TD3-BC-style behavior cloning on stored actions.
  • standard math Prior bounds from LOOP (Theorem 1) and Bertsekas (Lemma 6.1) are correct and applicable (Appendix A).
    Theorem 3.1 and Theorem 4.1 rely on these cited results; no verification is provided within this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint." pith.science (2026). https://pith.science/paper/LGMCAM66

@misc{pith2026250203550,
  author       = {Pith},
  title        = {Pith review of: TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LGMCAM66}},
  note         = {Machine review of arXiv:2502.03550}
}
read the original abstract

Model-based reinforcement learning algorithms that combine model-based planning and learned value/policy prior have gained significant recognition for their high data efficiency and superior performance in continuous control. However, we discover that existing methods that rely on standard SAC-style policy iteration for value learning, directly using data generated by the planner, often result in \emph{persistent value overestimation}. Through theoretical analysis and experiments, we argue that this issue is deeply rooted in the structural policy mismatch between the data generation policy that is always bootstrapped by the planner and the learned policy prior. To mitigate such a mismatch in a minimalist way, we propose a policy regularization term reducing out-of-distribution (OOD) queries, thereby improving value learning. Our method involves minimum changes on top of existing frameworks and requires no additional computation. Extensive experiments demonstrate that the proposed approach improves performance over baselines such as TD-MPC2 by large margins, particularly in 61-DoF humanoid tasks. View qualitative results at https://darthutopian.github.io/tdmpc_square/.

Figures

Figures reproduced from arXiv: 2502.03550 by the authors.

Figure 1
Figure 1. Value approximation error for TD-MPC2. The true value is estimated using the average dis￾counted return over 100 episodes following the nom￾inal policy π; Function estimation is obtained by Vˆ = Eπ[Qˆ]. The results are averaged over three seeds for an unbiased assessment. truth. This persistent value overestimation is also reflected in performance. According to benchmarking results [33], TD-MPC2 failed to acquire pe… view at source ↗
Figure 2
Figure 2. Toy Example [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Humanoid-Bench Locomotion Suite. Average episode return of our method (TD-M(PC)2 ) and baselines. We report mean performance and 95% CIs across 14 humanoid locomotion tasks. We do not include Reach-v0 in the average result due to its distinct reward scale. without any task-specific tuning. This consistency allows us to directly assess the adaptability and robustness of our approach across different tasks and environ… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: DM Control Suite. Average episode return of our method (TD-M(PC)2 ) and baselines. We report mean performance and 95% CIs across 7 high-dimensional continuous control tasks. We also present the average performance on all algorithms. 0 25 50 75 100 125 150 175 200 Value…
Figure 6
Figure 6. Figure 6: Ablation study on β. We evaluate all variants on two high-dimensional tasks from different domains: dog-trot and h1hand-run-v0. The results indi￾cate that our method is not sensitive to β. analysis, we demonstrate that standard policy iteration leads to compounding err…
Figure 7
Figure 7. Figure 7: TD-MPC2 ablation results of horizon at h1hand-run-v0. (a) Value estimation error with different horizons; (b) Episode return with different horizons. Variant with longer horizon shows convergent error growth pattern and better performance with more training steps. Mode…
Figure 8
Figure 8. Figure 8: Constrained policy update through AWAC Mean and 95% CIs across 3 ramdom seeds on high-dimensional tasks. Moreover, in scenarios where exploration is critical, intuitively we do not recommend employing conservative Q-learning 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visualizations. We demonstrate trajectories generated by our method 6 tasks across two benchmarks (DMControl and HumanoidBench) as qualitative results; tasks are listed as follows: Hopper-stand (A ∈ R 4 ), Humanoid-run (A ∈ R 21), Dog-run (A ∈ R 36), h1hand-run-v0 (A ∈…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion

    cs.RO 2025-06 conditional novelty 6.0 of 10

    DoublyAware combines conformal trajectory filtering with a group-relative policy constraint to improve sample efficiency of TD-MPC for simulated humanoid locomotion.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Argenson and G

    A. Argenson and G. Dulac-Arnold. Model-based of- fline planning. arXiv preprint arXiv:2008.05556, 2020

    arXiv 2008

  2. [2]

    Bertsekas

    D. Bertsekas. Neuro-dynamic programming. Athena Scientific, 1996. 8

    work page 1996

  3. [3]

    Bhardwaj, A

    M. Bhardwaj, A. Handa, D. Fox, and B. Boots. In- formation theoretic model predictive q-learning. In Learning for Dynamics and Control, pages 840–850. PMLR, 2020

    work page 2020

  4. [4]

    A. Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mah- mood, and M. White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23(253):1–79, 2022

    work page 2022

  5. [5]

    K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018

    2018

  6. [6]

    Fujimoto and S

    S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural in- formation processing systems, 34:20132–20145, 2021

    work page 2021

  7. [7]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing func- tion approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    work page 2018

  8. [8]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052–2062. PMLR, 2019

    work page 2019

Show all 41 references
  1. [9]

    D. Garg, J. Hejna, M. Geist, and S. Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023

    2023 arXiv

  2. [10]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep rein- forcement learning with a stochastic actor. In Inter- national conference on machine learning, pages 1861–

  3. [11]

    Hafner, T

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR, 2019

    2019

  4. [12]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020

    2010 arXiv

  5. [13]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mas- tering diverse domains through world models. arXiv preprint arXiv: 2301.04104, 2023

    2023 arXiv

  6. [14]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  7. [15]

    Hansen, X

    N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955, 2022

    2022 arXiv

  8. [16]

    Hansen-Estruch, I

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  9. [17]

    Janner, J

    M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Kabzan, L

    J. Kabzan, L. Hewing, A. Liniger, and M. N. Zeilinger. Learning-based model predictive control for autonomous racing. IEEE Robotics and Automa- tion Letters, 4(4):3363–3370, 2019

    2019

  11. [19]

    Katsigiannis and N

    S. Katsigiannis and N. Ramzan. Dreamer: A database for emotion recognition through eeg and ecg signals from wireless low-cost off-the-shelf devices. IEEE journal of biomedical and health informatics, 22(1):98– 107, 2017

    2017

  12. [20]

    Kostrikov, A

    I. Kostrikov, A. Nair, and S. Levine. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  13. [21]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Sta- bilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019

    2019

  14. [22]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020

    2020

  15. [23]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  16. [24]

    C. Li, A. Krause, and M. Hutter. Robotic world model: A neural network simulator for robust policy optimiza- tion in robotics. arXiv preprint arXiv:2501.10100 , 2025

    2025

  17. [25]

    Littman and A

    M. Littman and A. Moore. Reinforcement learning: A survey, journal of artificial intelligence research 4, 1996

    1996

  18. [26]

    Lowrey, A

    K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learn- ing and exploration via model-based control. arXiv preprint arXiv:1811.01848, 2018. 9

    2018 arXiv

  19. [27]

    Y . Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 IEEE/RSJ International Confer- ence on Intelligen...

    2023

  20. [28]

    R. Munos. Performance bounds in l p-norm for ap- proximate value iteration. SIAM journal on control and optimization, 46(2):541–561, 2007

    2007

  21. [29]

    A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  22. [30]

    Nakamoto, S

    M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y . Ma, C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Process- ing Systems, 36, 2024

    2024

  23. [31]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  24. [32]

    Schulman

    J. Schulman. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  25. [33]

    Sferrazza, D.-M

    C. Sferrazza, D.-M. Huang, X. Lin, Y . Lee, and P. Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipula- tion. arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  26. [34]

    Sikchi, W

    H. Sikchi, W. Zhou, and D. Held. Learning off-policy with online planning. InConference on Robot Learning, pages 1622–1633. PMLR, 2022

    2022

  27. [35]

    S. P. Singh and R. C. Yee. An upper bound on the loss from approximate optimal-value functions. Machine Learning, 16:227–233, 1994

    1994

  28. [36]

    R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160–163, 1991

    1991

  29. [37]

    R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  30. [38]

    Tassa, Y

    Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  31. [39]

    Thrun and A

    S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceed- ings of the 1993 connectionist models summer school, pages 255–263. Psychology Press, 2014

    1993

  32. [40]

    Williams, N

    G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1714–1721. IEEE, 2017

    2017

  33. [41]

    G. Zhou, S. Swaminathan, R. V . Raju, J. S. Guntupalli, W. Lehrach, J. Ortiz, A. Dedieu, M. L´azaro-Gredilla, and K. Murphy. Diffusion model predictive control. arXiv preprint arXiv:2410.05364, 2024. 10 A Theory and Discussion A.1 Useful Lemma Lemma A.1. [35] Suppose πk+1 is 1...

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.