Pith. sign in

REVIEW 3 major objections 4 minor 58 references

Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FLEXplore claims that a deliberately loose dynamics model, smoothed rewards, and a mutual-information exploration bonus reduce rollout regret in parameterized-action MDPs, with gains on six benchmarks.

desk verdict The algorithm is a plausible engineering combination, but the main theoretical contribution (Proposition 5.1) is mathematically invalid, so the exploration reward is ungrounded. read the letter →

arxiv 2501.02774 v1 pith:ZHUT47D7 submitted 2025-01-06 cs.LG

Zijian Wang , Bin Wang , Mingwen Shao , Hongbo Dou , Boxiang Tao This is my paper · ORCID
classification cs.LG
keywords parameterizedactionMDPsmodel-basedreinforcementlearningWassersteinmetricrewardsmoothingmutualinformationexplorationhybridspaceLipschitzcontinuitymodelpredictivepathintegral
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FLEXplore is a model-based reinforcement learning algorithm for parameterized-action MDPs (PAMDPs), where each discrete action carries continuous parameters. It argues that a deliberately 'loose' dynamics model, learned with a Wasserstein-based loss rather than exact state matching, preserves exploration while still controlling long-horizon error. On top of that, it smooths rewards around high-reward states and adds an auxiliary reward derived from a variational lower bound on the mutual information between states and hybrid actions. The paper claims this combination improves both early learning efficiency and asymptotic performance, and provides a Lipschitz-conditioned regret bound for rollout trajectories. A sympathetic reader would take away that model flexibility, not accuracy, is the key to sample-efficient hybrid-action RL.

What carries the argument

The load-bearing object is the flexible-dynamics loss $\mathcal{L}_{\mathrm{ex}}(\tau;f) = \max_{f: K_{d_R,d_R}\le 1}\sum_{t=t_0}^{t_0+H}\gamma^{t-t_0} |\mathbb{E}_{s_{t+1}} f(s_{t+1}) - \mathbb{E}_{\tau} f(\hat{s}_{t+1})|$. By Wasserstein duality, this loss is simultaneously an upper bound and a lower bound on the Wasserstein distance between the environment transition $T(\cdot\mid s_t,k_t,z_{k_t})$ and the learned transition $T_\phi(\cdot\mid s_t,k_t,z_{k_t})$, so minimizing it controls model error while leaving the model 'loose.' Two auxiliary mechanisms do the exploration work: FGSM-generated perturbed states flatten the learned reward's high-value neighborhoods, and an auxiliary reward $r_{\mathrm{aux}} = \mathbb{E}[\log T_\phi(s'\mid s,k,z_k) - \log p_\theta(z_k\mid s,k)]$ is added during model rollouts to encourage the agent to seek informative next states.

What would settle it

Take one state $s$, one discrete action $k$, and two equiprobable parameter values $z_k$. Let the learned dynamics put all probability on a single next state $s'$ independent of $z_k$, and let $p_\theta(z_k\mid s,k)$ be uniform on the two values. Then $I(s';(k,z_k)\mid s)=0$ but the proposed bound's right-hand side equals $0 - \mathbb{E}[\log p_\theta(z_k\mid s,k)] = 1 > 0$, so the claimed variational lower bound is violated by direct calculation.

Watch

Extended reading notes

Core claim

The paper's central claim is that a model-based RL agent for parameterized-action MDPs can improve both sample efficiency and asymptotic performance by deliberately not fitting the environment dynamics too tightly. FLEXplore's dynamics loss, a maximum over 1-Lipschitz test functions of the discounted mismatch between predicted and observed next states, is shown to sandwich the Wasserstein distance (earth mover's distance) between learned and true transitions; minimizing it therefore shrinks the one-step and H-step dynamics error without overfitting. Reward smoothing with FGSM-generated perturbations is applied only after the reward model stabilizes, expanding the neighborhood of high-reward states and lowering the local Lipschitz constant of the reward. An auxiliary reward built from $\log T_\phi(s'\mid s,k,z_k) - \log p_\theta(z_k\mid s,k)$ is claimed to be a variational lower bound on the mutual information between the next state and the hybrid action, guiding exploration during model rollouts. The paper reports that this combination attains the best early-training return on all six benchmarks and the best asymptotic return on four of them.

Load-bearing premise

The whole exploration bonus rests on a claimed inequality that the auxiliary reward is a true lower bound on how much the next state reveals about the chosen action. The proof of that inequality mixes entropies of different variables; if the inequality is actually false, the exploration bonus is just a heuristic.

Editorial extensions

If this is right

  • Minimizing $\mathcal{L}_{\mathrm{ex}}$ directly shrinks a term in the regret upper bound of Theorem 6.1, so better dynamics flexibility translates into tighter trajectory regret.
  • The loose-yet-flexible objective should make model-based agents more robust to distribution shift and adversarial perturbations in the transition dynamics, as the Adversarial Sample Test suggests.
  • Delaying reward smoothing until a stable training phase is important; applying it from the start hurts performance, implying a two-phase schedule for smoothing-based MBRL.
  • If the mutual-information bound held, the auxiliary reward would be a principled way to prevent policy collapse to deterministic local optima; the paper's position plots on Platform and Goal are consistent with that.
  • Empirically, FLEXplore attains the best early-training return on all six benchmarks and best asymptotic return on four of six.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Wasserstein-based looseness mechanism is metric-specific: replacing the Wasserstein distance with KL or total variation would make the loss insensitive to non-overlapping distributions, so a sliced-Wasserstein or maximum-mean-discrepancy variant is a natural testable alternative.
  • The exploration bonus likely needs a corrected derivation: a proper variational bound would condition on the entropy of the parameter given the next state, not on the entropy of the next state, so the exact form of $r_{\mathrm{aux}}$ should be revisited before relying on it theoretically.
  • The Hard move(8) result suggests an adaptive smoothing rule: only smooth positive maximum rewards; this is a cheap ablative experiment the paper does not run.
  • Although the paper targets hybrid action spaces, the loose-dynamics loss and its regret sandwich apply to any continuous-state MBRL problem, so the method generalizes beyond PAMDPs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FLEXplore, a model-based RL algorithm for parameterized-action MDPs (PAMDPs). It combines a Wasserstein-based dynamics loss (Eq. 7), an FGSM-based reward-smoothing loss (Eq. 16), and an auxiliary reward derived from a claimed variational lower bound on the mutual information between the next state and the hybrid action (Eqs. 19-23). The authors claim to prove that FLEXplore reduces the regret of rollout trajectories under Lipschitz conditions, and they report experiments on six PAMDP benchmarks comparing FLEXplore with DLPA, HyAR, HPPO, PDQN, and PATD3.

Significance. If the central results were correct, the paper would offer a useful MBRL recipe for hybrid action spaces: a distributional model loss with a theoretical link to the Wasserstein metric, a smoothing mechanism with an ablation study, and an exploration bonus. The Wasserstein analysis in Theorems 3.1 and 3.2 is a legitimate application of Kantorovich-Rubinstein duality, and Corollary 3.1 goes through because the supremum is moved inside the integral in the valid direction. However, the central theoretical result in Section V is mathematically false, and the regret-reduction claim in Section VI is asserted rather than derived. Because these two claims are load-bearing for the abstract and the contributions, the paper as submitted does not establish its main claims. The empirical study is broad (six benchmarks, ablations) but lacks statistical testing, and Section VII.D provides only qualitative evidence for the MI exploration mechanism.

major comments (3)
  1. [V, Eqs. (19)-(21)] Proposition 5.1 is mathematically incorrect. The chain rule gives I(s'; z_k | k, s) = H(z_k | k, s) - H(z_k | s', k, s) = H(s' | k, s) - H(s' | z_k, k, s), not H(z_k | s, k, theta) - H(s' | s, k, z_k, phi). The quantity in Eq. (20) subtracts entropies of two different variables and is not a mutual information. Consequently the 'variational lower-bound' in Eq. (21) does not lower-bound the target MI, and the auxiliary reward r_aux in Eq. (23) has no theoretical grounding as an MI-maximizing exploration signal. A concrete counterexample shows the claimed inequality is false: for scalar variables, take Z ~ N(0, 10^4), S' = 10^{-3} Z + epsilon with epsilon ~ N(0,1), p_theta = N(0, 10^4), and T_phi = N(10^{-3} Z, 1). Then I(S'; Z) = 0.5 log(1 + 10^{-2}) ≈ 0.005, while E[log T_phi - log p_theta] ≈ 4.6, violating Eq. (19). This invalidates the theoretical basis for the exploration enhancement claimed in Section V and the corresponding claim in the abstract.
  2. [VI, Theorem 6.1 and Eq. (26)] The regret-reduction claim is asserted rather than derived. Theorem 6.1 is quoted from DLPA, and the only argument given is that Lex is an upper bound on W and that Eq. (16) reduces a local Lipschitz constant. Reducing an upper bound does not by itself establish that FLEXplore's actual regret is lower, and no comparison with DLPA's bound is made. Moreover, Eq. (26) only states the monotonicity of min after assuming the local Lipschitz constants are ordered; the paper does not prove that the smoothing loss in Eq. (16) reduces L^S_R. Thus the abstract's claim that FLEXplore 'can reduce the regret of the rollout trajectory' is unsupported.
  3. [VII.D and Table I] The empirical support for the two main contributions is incomplete. Section VII.D reports only scatter plots and heat maps for the MI reward, with no quantitative MI estimate and no significance test; this cannot compensate for the failure of Proposition 5.1. In Table I, on Goal the HyAR baseline achieves a higher asymptotic return (34.12 ± 2.99 vs 32.21 ± 3.07), and on Hard move(8) DLPA achieves a higher return (7.22 ± 8.21 vs 6.22 ± 6.10). With only four seeds and no statistical testing, the claim of 'outstanding asymptotic performance' on the benchmarks is not established.
minor comments (4)
  1. [II.A, Eq. (2)] The Lipschitz-constant subscripts in Eq. (2) appear to be swapped: the second inequality should use L^K_T with d_K and the third should use L^Z_T with d_Z.
  2. [Throughout] There are numerous typos, including 'Lipstchiz', 'losse', 'effciency', and 'Addtionally'; a careful proofread is needed.
  3. [IV, Eq. (17)] The FGSM formula in Eq. (17) should specify whether the gradient is taken with respect to s-tilde at s-tilde = s' and how the perturbation is re-projected if needed; the notation is ambiguous.
  4. [Fig. 4] Figure 4 reports averages over four seeds without error bars or shaded regions; adding variability information would make the early-learning-efficiency claim easier to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the main theoretical claims rest on external Wasserstein duality and the DLPA regret theorem, while the flawed MI bound in Proposition 5.1 is a correctness error rather than a self-referential derivation.

full rationale

I reviewed the derivation chain from the designed loss Lex (Eq. 7) through Theorems 3.1-3.2, Corollaries 3.1-3.2, Proposition 5.1, and Theorem 6.1. The loss Lex is deliberately constructed from the Kantorovich-Rubinstein dual of the Wasserstein metric, so Theorems 3.1-3.2 are bounding relationships implied by that construction rather than hidden predictions; they do not fit any parameter to the outcome they claim to explain. The regret bound in Theorem 6.1 is explicitly quoted from the external DLPA work, not from the authors' own prior results: the paper states 'Theorem 6.1 (the regret upper bound, proved by [19])' and its proof says 'See [19] for the remaining proof.' Inserting the authors' Lex into this external bound is transparent reuse of an independent result, not circularity. The only self-citation is [51] in a list of beam-search references in Section III.C, and it is not load-bearing. The paper does contain a serious mathematical flaw that I flag per the review rule: Proposition 5.1 claims I(s';(k,z_k)|s,Φ) >= E[log T_phi(s'|s,k,z_k) - log p_theta(z_k|s,k)], but the proof reduces the conditional mutual information to H(z_k|s,k,theta) - H(s'|s,k,z_k,phi), which subtracts entropies of different variables and is not a mutual information. Eq. (21) at best bounds a different quantity, and r_aux in Eq. (23) therefore lacks the claimed variational-MI grounding. This is a correctness or validity problem in the derivation, not a circularity: the bound is attempted through an explicit information-theoretic proof and fails on its own terms, rather than being equivalent to its input by construction. No fitted quantity is relabeled as a prediction, no uniqueness theorem is imported from the authors, and no load-bearing self-citation is present, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on Lipschitz structure inherited from DLPA, a standard duality argument for the loss bounds, and an unproven smoothing-to-Lipschitz step. The mutual-information auxiliary reward is the weakest link: the claimed variational bound is not mathematically justified. No new physical or conceptual entities are introduced.

free parameters (6)
  • lambda = 0.3-0.7 per environment (Table III)
    Weight of the flexible-critic loss Lex in the total dynamics loss; chosen per benchmark.
  • mu = 0.5 (all environments)
    Weight of the reward-smoothing loss Lsmt.
  • eta = 0.01 for Platform/Goal/Catch, 0.05 for Hard move(4/6/8)
    Weight of the auxiliary mutual-information reward.
  • epsilon = 0.1 for Platform/Goal/Catch, 0.3 for Hard move(4/6), 0.5 for Hard move(8)
    Perturbation bound for FGSM reward smoothing.
  • T = 10,000 (Platform/Goal/Catch), 50,000 (Hard move(4)), 100,000 (Hard move(6/8))
    Threshold after which reward smoothing is enabled.
  • H = 8 (Platform/Goal), 5 (others)
    Model rollout horizon in the loss and planning.
assumptions (5)
  • domain assumption The PAMDP is Lipschitz continuous (Definition 2.2), with finite Lipschitz constants for dynamics and reward.
    Every theorem in the paper relies on this smoothness assumption; the regret bound additionally requires L^S_T < 1.
  • standard math Kantorovich-Rubinstein duality expresses Wasserstein distance as a supremum over 1-Lipschitz functions.
    Used in Theorem 3.1, 3.2, and Corollary 3.1; standard result from optimal transport.
  • standard math Composition Lemma for Wasserstein distance of composed transitions (Asadi et al. 2018).
    Used in Corollary 3.2 to propagate errors over H steps.
  • standard math Variational lower bound on mutual information (Barber-Agakov).
    Invoked in Proposition 5.1, but the application is incorrect because the entropy terms are mismatched.
  • ad hoc to paper The learned reward's local Lipschitz constant can be reduced by the smoothing loss (Eq. 16), and this lowers the global Lipschitz constant in Eq. (26).
    This is asserted with a citation to [55]; no proof is given, and it is used to claim regret reduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes." pith.science (2026). https://pith.science/paper/ZHUT47D7

@misc{pith2026250102774,
  author       = {Pith},
  title        = {Pith review of: Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHUT47D7}},
  note         = {Machine review of arXiv:2501.02774}
}
read the original abstract

Hybrid action models are widely considered an effective approach to reinforcement learning (RL) modeling. The current mainstream method is to train agents under Parameterized Action Markov Decision Processes (PAMDPs), which performs well in specific environments. Unfortunately, these models either exhibit drastic low learning efficiency in complex PAMDPs or lose crucial information in the conversion between raw space and latent space. To enhance the learning efficiency and asymptotic performance of the agent, we propose a model-based RL (MBRL) algorithm, FLEXplore. FLEXplore learns a parameterized-action-conditioned dynamics model and employs a modified Model Predictive Path Integral control. Unlike conventional MBRL algorithms, we carefully design the dynamics loss function and reward smoothing process to learn a loose yet flexible model. Additionally, we use the variational lower bound to maximize the mutual information between the state and the hybrid action, enhancing the exploration effectiveness of the agent. We theoretically demonstrate that FLEXplore can reduce the regret of the rollout trajectory through the Wasserstein Metric under given Lipschitz conditions. Our empirical results on several standard benchmarks show that FLEXplore has outstanding learning efficiency and asymptotic performance compared to other baselines.

Figures

Figures reproduced from arXiv: 2501.02774 by the authors.

Figure 1
Figure 1. For simplicity, we designate the state as [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An intuitional example of reward smoothing for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The framework of FLEXplore. In the model learning phase (the left subplot), the agent samples sequence τ = {s0, k0, zk0 , k1, zk1 , . . . , kH, zkH } from the replay buffer and then employs it within the model to generate predicted states sˆt+1 and rewards rˆt for each timestep t. It is trained with Ltotal dyn and Ltotal rew respectively, which correspond to flexible model learning in Section 3 and reward smoothing … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance evaluation of FLEXplore and baselines over 4 seeds across six standard PAMDP benchmarks. To demonstrate the agent’s learning [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Investigation on loose yet flexible dynamics learning. (a) Dynamics [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Investigation on reward smoothing mechanism. (a) Performance [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The visualization of the agent/ball’s position at each time step on the [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 33 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. , “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  2. [2]

    Continuous control with deep reinforcement learning,

    T. Lillicrap, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015

    arXiv 2015

  3. [3]

    Trust region policy optimization,

    J. Schulman, “Trust region policy optimization,” arXiv preprint arXiv:1502.05477, 2015

    arXiv 2015

  4. [4]

    Deep rein- forcement learning framework for autonomous driving,

    A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani, “Deep rein- forcement learning framework for autonomous driving,” arXiv preprint arXiv:1704.02532, 2017

    arXiv 2017

  5. [5]

    Continuous mdp homomorphisms and homomorphic policy gradient,

    S. Rezaei-Shoshtari, R. Zhao, P. Panangaden, D. Meger, and D. Precup, “Continuous mdp homomorphisms and homomorphic policy gradient,” Advances in Neural Information Processing Systems, vol. 35, pp. 20189– 20204, 2022

    work page 2022

  6. [6]

    Td-mpc2: Scalable, robust world models for continuous control,

    N. Hansen, H. Su, and X. Wang, “Td-mpc2: Scalable, robust world models for continuous control,” arXiv preprint arXiv:2310.16828, 2023

    arXiv 2023

  7. [7]

    Movie: Visual model-based policy adaptation for view generalization,

    S. Yang, Y . Ze, and H. Xu, “Movie: Visual model-based policy adaptation for view generalization,” Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  8. [8]

    Making better decision by directly planning in continuous control,

    J. Zhu, Y . Wang, L. Wu, T. Qin, W. Zhou, T.-Y . Liu, and H. Li, “Making better decision by directly planning in continuous control,” in The Eleventh International Conference on Learning Representations , 2021

    work page 2021

Show all 58 references
  1. [9]

    Hierarchical advantage for reinforcement learning in parameterized action space,

    Z. Hu and T. Kaneko, “Hierarchical advantage for reinforcement learning in parameterized action space,” in 2021 IEEE Conference on Games (CoG), pp. 1–8, IEEE, 2021

    2021

  2. [10]

    Deep reinforcement learning in parame- terized action space,

    M. Hausknecht and P. Stone, “Deep reinforcement learning in parame- terized action space,” arXiv preprint arXiv:1511.04143 , 2015

    2015 arXiv

  3. [11]

    Deep multi-agent reinforcement learning with discrete-continuous hybrid action spaces,

    H. Fu, H. Tang, J. Hao, Z. Lei, Y . Chen, and C. Fan, “Deep multi-agent reinforcement learning with discrete-continuous hybrid action spaces,” arXiv preprint arXiv:1903.04959 , 2019

    1903 arXiv

  4. [12]

    Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,

    J. Xiong, Q. Wang, Z. Yang, P. Sun, L. Han, Y . Zheng, H. Fu, T. Zhang, J. Liu, and H. Liu, “Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,” arXiv preprint arXiv:1810.06394 , 2018

    arXiv

  5. [13]

    Hybrid actor-critic rein- forcement learning in parameterized action space,

    Z. Fan, R. Su, W. Zhang, and Y . Yu, “Hybrid actor-critic rein- forcement learning in parameterized action space,” arXiv preprint arXiv:1903.01344, 2019

    1903 arXiv

  6. [14]

    Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation,

    B. Li, H. Tang, Y . Zheng, J. Hao, P. Li, Z. Wang, Z. Meng, and L. Wang, “Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation,” arXiv preprint arXiv:2109.05490 , 2021

    2021 arXiv

  7. [15]

    Reinforcement learning with parameterized actions,

    W. Masson, P. Ranchod, and G. Konidaris, “Reinforcement learning with parameterized actions,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , pp. 1934–1940, AAAI Press, 2016

    1934

  8. [16]

    Dream to control: Learn- ing behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learn- ing behaviors by latent imagination,” arXiv preprint arXiv:1912.01603 , 2019

    1912 arXiv

  9. [17]

    Mastering atari with discrete world models,

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” arXiv preprint arXiv:2010.02193 , 2020

    2010 arXiv

  10. [18]

    Privileged sensing scaffolds reinforcement learning,

    E. S. Hu, J. Springer, O. Rybkin, and D. Jayaraman, “Privileged sensing scaffolds reinforcement learning,” arXiv preprint arXiv:2405.14853 , 2024

    2024 arXiv

  11. [19]

    Model-based rein- forcement learning for parameterized action spaces,

    R. Zhang, H. Fu, Y . Miao, and G. Konidaris, “Model-based rein- forcement learning for parameterized action spaces,” arXiv preprint arXiv:2404.03037, 2024

    2024 arXiv

  12. [20]

    The benefits of model-based generalization in reinforcement learning,

    K. Young, A. Ramesh, L. Kirsch, and J. Schmidhuber, “The benefits of model-based generalization in reinforcement learning,” arXiv preprint arXiv:2211.02222, 2022

    2022 arXiv

  13. [21]

    Diminishing return of value expansion methods in model-based reinforcement learning,

    D. Palenicek, M. Lutter, J. Carvalho, and J. Peters, “Diminishing return of value expansion methods in model-based reinforcement learning,” in ICLR, 2023

    2023

  14. [22]

    Models, pixels, and rewards: Evaluating design trade- offs in visual model-based reinforcement learning,

    M. Babaeizadeh, M. T. Saffar, D. Hafner, H. Kannan, C. Finn, S. Levine, and D. Erhan, “Models, pixels, and rewards: Evaluating design trade- offs in visual model-based reinforcement learning,” arXiv preprint arXiv:2012.04603, 2020

    2012 arXiv

  15. [23]

    Dreamsmooth: Improving model- based reinforcement learning via reward smoothing,

    V . Lee, P. Abbeel, and Y . Lee, “Dreamsmooth: Improving model- based reinforcement learning via reward smoothing,” arXiv preprint arXiv:2311.01450, 2023

    2023 arXiv

  16. [24]

    Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,

    C. J. Bester, S. D. James, and G. D. Konidaris, “Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,” arXiv preprint arXiv:1905.04388, 2019

    1905 arXiv

  17. [25]

    Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasks,

    X. Zhang, S. Jin, C. Wang, X. Zhu, and M. Tomizuka, “Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasks,” in 2022 International conference on robotics and automation (ICRA) , pp. 9881–9887, IEEE, 2022

    2022

  18. [26]

    Lipschitz continuity in model- based reinforcement learning,

    K. Asadi, D. Misra, and M. Littman, “Lipschitz continuity in model- based reinforcement learning,” in International Conference on Machine Learning, pp. 264–273, PMLR, 2018

    2018

  19. [27]

    Markov processes over denumerable products of spaces, describing large systems of automata,

    L. N. Vaserstein, “Markov processes over denumerable products of spaces, describing large systems of automata,” Problemy Peredachi Informatsii, vol. 5, no. 3, pp. 64–72, 1969

    1969

  20. [28]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014

    2014 arXiv

  21. [29]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal , vol. 27, no. 3, pp. 379–423, 1948

    1948

  22. [30]

    The im algorithm: a variational approach to information maximization,

    D. Barber and F. Agakov, “The im algorithm: a variational approach to information maximization,” Advances in neural information processing systems, vol. 16, no. 320, p. 201, 2004

    2004

  23. [31]

    Learning-based model predictive control for markov decision processes,

    R. R. Negenborn, B. De Schutter, M. A. Wiering, and H. Hellen- doorn, “Learning-based model predictive control for markov decision processes,” IFAC Proceedings Volumes , vol. 38, no. 1, pp. 354–359, 2005

    2005

  24. [32]

    Optimization of computer simulation models with rare events,

    R. Y . Rubinstein, “Optimization of computer simulation models with rare events,” European Journal of Operational Research , vol. 99, no. 1, pp. 89–112, 1997

    1997

  25. [33]

    Plan to predict: Learning an uncertainty-foreseeing model for model-based reinforce- ment learning,

    Z. Wu, C. Yu, C. Chen, J. Hao, and H. H. Zhuo, “Plan to predict: Learning an uncertainty-foreseeing model for model-based reinforce- ment learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 15849–15861, 2022

    2022

  26. [34]

    Choreographer: Learning and adapting skills in imagination,

    P. Mazzaglia, T. Verbelen, B. Dhoedt, A. Lacoste, and S. Rajeswar, “Choreographer: Learning and adapting skills in imagination,” arXiv preprint arXiv:2211.13350, 2022

    2022 arXiv

  27. [35]

    Mismatched no more: Joint model-policy optimization for model- based rl,

    B. Eysenbach, A. Khazatsky, S. Levine, and R. R. Salakhutdinov, “Mismatched no more: Joint model-policy optimization for model- based rl,” Advances in Neural Information Processing Systems , vol. 35, pp. 23230–23243, 2022

    2022

  28. [36]

    Differen- tiable mpc for end-to-end planning and control,

    B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter, “Differen- tiable mpc for end-to-end planning and control,” Advances in neural information processing systems , vol. 31, 2018

    2018

  29. [37]

    On information and sufficiency,

    S. Kullback and R. A. Leibler, “On information and sufficiency,” The annals of mathematical statistics , vol. 22, no. 1, pp. 79–86, 1951

    1951

  30. [38]

    Measures of distance between probability distributions,

    J. Chung, P. Kannappan, C. T. Ng, and P. Sahoo, “Measures of distance between probability distributions,” Journal of mathematical analysis and applications, vol. 138, no. 1, pp. 280–292, 1989

    1989

  31. [39]

    Dr. strategy: Model-based generalist agents with strategic dreaming,

    H. Hamed, S. Kim, D. Kim, J. Yoon, and S. Ahn, “Dr. strategy: Model-based generalist agents with strategic dreaming,” arXiv preprint arXiv:2402.18866, 2024. JOURNAL OF LATEX IEEE TRANS, VOL., NO., NOVEMBER 2024 14

    2024 arXiv

  32. [40]

    When to trust your model: Model-based policy optimization,

    M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” Advances in neural information pro- cessing systems, vol. 32, 2019

    2019

  33. [41]

    Model-based reinforcement learning via meta-policy optimization,

    I. Clavera, J. Rothfuss, J. Schulman, Y . Fujita, T. Asfour, and P. Abbeel, “Model-based reinforcement learning via meta-policy optimization,” in Conference on Robot Learning , pp. 617–629, PMLR, 2018

    2018

  34. [42]

    The virtues of laziness in model-based rl: A unified objective and algo- rithms,

    A. Vemula, Y . Song, A. Singh, D. Bagnell, and S. Choudhury, “The virtues of laziness in model-based rl: A unified objective and algo- rithms,” in International Conference on Machine Learning , pp. 34978– 35005, PMLR, 2023

    2023

  35. [43]

    Iterative value-aware model learning,

    A.-m. Farahmand, “Iterative value-aware model learning,” Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  36. [44]

    Model-based value expansion for efficient model-free rein- forcement learning,

    V . Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine, “Model-based value expansion for efficient model-free rein- forcement learning,” in Proceedings of the 35th International Conference on Machine Learning (ICML 2018) , 2018

    2018

  37. [45]

    Villani et al

    C. Villani et al. , Optimal transport: old and new , vol. 338. Springer, 2009

    2009

  38. [46]

    Generative adversarial nets,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” Advances in neural information processing systems , vol. 27, 2014

    2014

  39. [47]

    Towards principled methods for train- ing generative adversarial networks,

    M. Arjovsky and L. Bottou, “Towards principled methods for train- ing generative adversarial networks,” arXiv preprint arXiv:1701.04862 , 2017

    2017 arXiv

  40. [48]

    Wasserstein generative ad- versarial networks,

    M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative ad- versarial networks,” in International conference on machine learning , pp. 214–223, PMLR, 2017

    2017

  41. [49]

    Harpy, a connected speech recognition system,

    B. P. Lowerre and B. R. Reddy, “Harpy, a connected speech recognition system,” The Journal of the Acoustical Society of America , vol. 59, no. S1, pp. S97–S97, 1976

    1976

  42. [50]

    Google’s neural machine translation system: Bridging the gap between human and machine translation,

    Y . Wu, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  43. [51]

    Walk wisely on graph: Knowledge graph reasoning with dual agents via efficient guidance- exploration,

    Z. Wang, B. Wang, H. Jing, H. Li, and H. Dou, “Walk wisely on graph: Knowledge graph reasoning with dual agents via efficient guidance- exploration,” arXiv preprint arXiv:2408.01880 , 2024

    2024 arXiv

  44. [52]

    Spectral norm regularization for improving the generalizability of deep learning,

    Y . Yoshida and T. Miyato, “Spectral norm regularization for improving the generalizability of deep learning,” arXiv preprint arXiv:1705.10941, 2017

    2017 arXiv

  45. [53]

    Categorical reparameterization with gumbel-softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144 , 2016

    2016 arXiv

  46. [54]

    Introduction to online convex optimization,

    E. Hazan et al., “Introduction to online convex optimization,” Founda- tions and Trends® in Optimization , vol. 2, no. 3-4, pp. 157–325, 2016

    2016

  47. [55]

    Is model ensemble necessary? model-based rl via a single model with lipschitz regularized value function,

    R. Zheng, X. Wang, H. Xu, and F. Huang, “Is model ensemble necessary? model-based rl via a single model with lipschitz regularized value function,” arXiv preprint arXiv:2302.01244 , 2023

    2023 arXiv

  48. [56]

    Visualizing data using t-sne.,

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.,” Journal of machine learning research , vol. 9, no. 11, 2008

    2008

  49. [57]

    General boundary conditions for denumberable markov processes,

    E. B. Dynkin, “General boundary conditions for denumberable markov processes,” Theory of Probability & Its Applications , vol. 12, no. 2, pp. 187–221, 1967

    1967

  50. [58]

    I. S. Berezin and N. P. Zhidkov, Computing Methods: Adiwes Interna- tional Series in the Engineering Sciences , vol. 1. Elsevier, 2014

    2014

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.