Pith. sign in

REVIEW 1 major objections 3 minor 43 references

Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning

T0 review · 1 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that dynamic non-prehensile transport can be learned from 50–100 task-space demonstrations with sparse slip labels, and deployed safely by combining an ensemble of value functions with uncertainty-aware model-predictive…

desk verdict A solid empirical paper whose core claim holds up, but the novel-object generalization is oversold and the missing code/constants keep it short of an accept as-is. read the letter →

arxiv 2412.00086 v1 pith:LYJOXZNR submitted 2024-11-27 cs.RO cs.LG

Neel Jawale , Byron Boots , Balakumar Sundaralingam , Mohak Bhardwaj This is my paper · ORCID
classification cs.ROcs.LG
keywords non-prehensilemanipulationrobotwaitertaskmodel-predictivecontrolofflinereinforcementlearningvaluefunctionensemblesparserewardlabelsfromdemonstrationsslipavoidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that a robot can learn the waiter task—carrying an object on a tray without it sliding—from very few demonstrations and then handle objects it never saw. The proposed method trains a team of value functions from end-effector trajectories labeled only with sparse slip/no-slip outcomes, then uses them inside a model-predictive controller that treats disagreement among the team as a risk signal. If this works, it matters because current analytic controllers need accurate object friction and inertia, while behavior cloning needs many demonstrations; this recipe promises fast teaching with minimal labeling. On a real Franka Panda arm, the paper reports success on novel convex and household objects after 50–100 demonstrations, and shows it can learn from a demonstrator with wrong friction assumptions.

What carries the argument

The load-bearing object is an ensemble of value functions $V_{\phi_i}$ trained with the Bellman error objective of Eq. (1) on transitions $(x_t, c, x_{t+1})$, where $x$ is the end-effector pose, twist, and acceleration and $c \in \{0, c_{\text{friction}}\}$ is a sparse slip label. Online, MPC rolls out trajectories and scores them with a pessimistic return $\hat{G}_{\text{pess}} = \log\sum_i \exp(\hat{G}_i / \lambda)$ of Eq. (2), which downweights trajectories where ensemble members disagree; applying this pessimism only at the initial state rather than pointwise is what avoids the over-conservatism identified in prior offline RL work. The value functions are not goal-conditioned: goal reaching comes from an $L^2$ cost, while the learned term supplies long-horizon slip avoidance.

What would settle it

Train a value-function ensemble on a cube with a fixed surface friction, then test on objects that produce identical end-effector trajectories and nominal friction but differ in internal mass distribution or contact-patch roughness; if success rates collapse while end-effector kinematics are held fixed, the Markov-state assumption in Eq. (1) is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the robot waiter task can be solved by offline value-function learning combined with online model-predictive control, without ever giving the learner the object's physical properties or dense costs. An ensemble of value functions is trained on end-effector states and sparse slip/no-slip labels, and the controller's trajectory-return estimate is made conservative through a log-sum-exp aggregation over the ensemble. This conservative value MPC (CV-MPC) is reported to match a friction-aware MPC demonstrator with exact object knowledge, beat the same demonstrator when its friction estimate is biased, and generalize to eight convex test objects and five household objects on a real Franka Panda arm, with success rates of about 73–95% and 80% or above respectively.

Load-bearing premise

The learned value functions see only the end-effector's motion and a sparse slip/no-slip label, so the whole method assumes these observations are enough to predict when an object will slip; if slip depends on object-specific properties such as contact patch, surface roughness, or internal mass distribution that the end-effector kinematics do not capture, transfer to novel objects will fail.

Editorial extensions

If this is right

  • A practitioner with only end-effector demonstrations and sparse failure labels can deploy dynamic manipulation on hardware at 50 Hz using a single GPU, without estimating object friction or inertia.
  • The method can improve over a suboptimal demonstrator: when the demonstrator uses an incorrect friction coefficient, the learned value functions still recover higher success rates.
  • Learned value functions transfer to unseen objects: real-world tests report 73–95% success on eight convex shapes after training on a cube, and 80% or above on household objects after training on five hard convex objects.
  • The end-effector rotation is the critical observation; rotation alone outperforms full position-plus-rotation when training and test start positions differ, suggesting the learned skill is about tilt and motion, not absolute location.
  • Because the approach integrates as a cost term in off-the-shelf MPC, it can inherit constraint satisfaction, smooth joint-space motion, and the optimizer's safety properties.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the core assumption holds, the same recipe should extend to other contact-rich dynamic skills—such as carrying open containers, sweeping, or pushing—where end-effector kinematics plus sparse failure labels carry enough information; the paper only demonstrates the waiter task.
  • Human teleoperation would likely work as a demonstration source, since the learning pipeline never needs expert actions or dense labels; the paper tests only with an algorithmic demonstrator.
  • A direct stress test of the Markov assumption would hold end-effector trajectories fixed while varying object surface roughness or internal mass distribution; if slip prediction degrades sharply, the value-function state must be augmented with object properties.
  • The log-sum-exp pessimism over the ensemble is one possible uncertainty aggregator; alternatives such as max or quantile aggregation could be compared without changing the rest of the framework.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. The paper proposes Conservative Value MPC (CV-MPC), a method for the 'robot waiter' non-prehensile transport task that trains an ensemble of state-value functions from 50-100 task-space demonstrations labeled only with sparse slip/no-slip costs, then combines the ensemble's return estimates pessimistically via a log-sum-exp objective inside a sampling-based MPC (STORM). The authors evaluate the method in simulation and on a Franka Panda, reporting high success rates on novel convex and household objects, ablations over ensemble size, pessimism temperature, observation spaces, and pessimism schemes, and a simulation study showing improvement over a deliberately biased-friction demonstrator.

Significance. If the claims hold, the paper is practically significant: it demonstrates that a dynamic, contact-rich manipulation skill can be learned from a modest number of end-effector-only demonstrations without explicit object models, and it integrates cleanly with off-the-shelf MPC. The empirical effort is substantial, with over 800 real-world trials across 13 objects, and the paper includes useful ablations, especially the observation-space study (Table III), the initial-state versus pointwise pessimism comparison (Fig. 8), and the biased-friction demonstrator experiment (Fig. 4). The main weaknesses are that the central generalization claim rests on an undefended state-sufficiency assumption, the real-world observation representation is not fully pinned down, and success rates are reported without confidence intervals or significance tests, making several comparative claims hard to evaluate.

major comments (1)
  1. [Sec. II / Eq. (1) and Sec. VI-B]
minor comments (3)
  1. [Eq. (2)] The notation for the pessimistic return is inconsistent: the definition above Eq. (2) uses h in H and u in H, while the equation writes the log-sum-exp over ensemble members, but the relationship between per-member return, the running-cost sum, and the terminal value is not made explicit. Please clarify whether the log-sum-exp is over per-step value sums or terminal values, and make the role of lambda explicit.
  2. [Sec. IV-B] There is a typo in '60 trails' (should be '60 trials') in the description of Case Study 2, and the total trial count is described as 'over 800 trials' while the reported case studies sum to 940 real-world trials; please reconcile the count.
  3. [Sec. VI-C1] The observation ablation is informative, but the four observation sets should be described in a single consistent notation; currently 'ofull_t', 'xfull_t', and 'wT_ee' are used interchangeably, which makes it hard to map the ablation back to the deployed system.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: value functions are fit to external slip labels and evaluated on held-out trials and objects; self-citations are non-load-bearing.

full rationale

The paper's central derivation chain is not circular. The learned value functions are trained with Eq. (1) on task-space transitions whose sparse labels are external slip/no-slip observations from friction-cone violation or measured slip distance, not on the quantities later reported as results. The online MPC minimizes G = G_STORM + G_pess, where G_pess is a log-sum-exp ensemble return from Eq. (2); this is a control objective, not a re-statement of the training labels, and success is measured by independent kinematic and slip criteria. The 'improving over suboptimal demonstrator' experiment uses trajectories from a demonstrator with a biased friction coefficient but labels true slip, so the value function can genuinely exploit the demonstrator's failures rather than merely reproducing its inputs. The self-citations present (STORM [6], blending [10], offline-from-observations [18]) are used as tools or related work, not as uniqueness theorems or as justification that the predicted result holds by construction; STORM is an external, code-released MPC framework. The concern that Eq. (1)'s end-effector-only state may not be sufficient for cross-friction generalization is a robustness and assumption limitation, not a circular reduction, and the paper explicitly keeps mu fixed in Case Study 1 and uses 100 demonstrations in Case Study 2. No step reduces by construction to its own input, so the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper uses standard RL and MPC machinery, but it does not report several constants that the central results depend on (gamma, H, c_friction). The most load-bearing domain assumption is that end-effector kinematics plus sparse slip labels are a sufficient state for predicting slip risk; the observation ablation supports this only for rotation, with full position-dependent observations failing when the start location changes.

free parameters (5)
  • Pessimism temperature lambda = lambda = 20 (selected in simulation ablation, Fig. 2)
    Controls how sharply the log-sum-exp combines ensemble value predictions; the paper chooses the best-performing value from a sweep.
  • Ensemble size K = K = 80 (selected in simulation ablation, Fig. 2)
    Number of value networks; chosen from the ablation because K = 80 with lambda = 20 gave the best success.
  • Discount factor gamma = not reported
    Appears in the Bellman residual (Eq. 1) and in return definitions; its value is never stated.
  • Failure cost c_friction = not reported
    Magnitude of the sparse slip label in Eq. (1); it sets the scale of all learned values and therefore affects MPC weights.
  • MPC horizon H = not reported
    Finite lookahead horizon in the STORM MPC rollouts; never specified.
assumptions (5)
  • domain assumption End-effector state x_r = [pose, twist, acceleration] is a sufficient Markov state for predicting future slip risk.
    The value function is trained only on these observations (Eq. 1), and the observation ablations in Sec. VI-C.1 show that omitting orientation Re collapses success to 5 percent, implying the state choice is load-bearing.
  • domain assumption The tray is rigidly grasped in a fixed known pose relative to the gripper, so end-effector motion fully determines the platform the object sits on.
    Stated in Sec. II; without this, end-effector state would not determine the base motion of the object.
  • domain assumption MPC has a sufficiently accurate robot model to predict end-effector states from commanded joint accelerations.
    Assumed in Sec. III-B: the paper states MPC has a sufficiently accurate robot model to predict end-effector states resulting from applied controls.
  • domain assumption The sparse slip/no-slip labels are a reliable proxy for the true friction-cone constraint cost, and c_friction is a valid cost for all objects.
    Labels come from human monitoring in the real world or automatic slip distance in simulation; the value functions inherit all label noise.
  • ad hoc to paper The log-sum-exp over ensemble return estimates (Eq. 2) provides a useful pessimistic estimate for out-of-distribution states.
    No formal guarantee is given; it is motivated by offline RL pessimism and evaluated empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning." pith.science (2026). https://pith.science/paper/LYJOXZNR

@misc{pith2026241200086,
  author       = {Pith},
  title        = {Pith review of: Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYJOXZNR}},
  note         = {Machine review of arXiv:2412.00086}
}
read the original abstract

We investigate the problem of teaching a robot manipulator to perform dynamic non-prehensile object transport, also known as the `robot waiter' task, from a limited set of real-world demonstrations. We propose an approach that combines batch reinforcement learning (RL) with model-predictive control (MPC) by pretraining an ensemble of value functions from demonstration data, and utilizing them online within an uncertainty-aware MPC scheme to ensure robustness to limited data coverage. Our approach is straightforward to integrate with off-the-shelf MPC frameworks and enables learning solely from task space demonstrations with sparsely labeled transitions, while leveraging MPC to ensure smooth joint space motions and constraint satisfaction. We validate the proposed approach through extensive simulated and real-world experiments on a Franka Panda robot performing the robot waiter task and demonstrate robust deployment of value functions learned from 50-100 demonstrations. Furthermore, our approach enables generalization to novel objects not seen during training and can improve upon suboptimal demonstrations. We believe that such a framework can reduce the burden of providing extensive demonstrations and facilitate rapid training of robot manipulators to perform non-prehensile manipulation tasks. Project videos and supplementary material can be found at: https://sites.google.com/view/cvmpc.

Figures

Figures reproduced from arXiv: 2412.00086 by the authors.

Figure 1
Figure 1. The CV-MPC framework applied to the robot waiter problem. (left) In the offline phase, we collect end-effector only demonstrations of transporting object placed on the tray and label transitions with costs indicating slip (c = cfriction) or no-slip (c = 0). We then train an ensemble of neural networks to independently predict the value function. (right) During online deployment, we use the pretrained value ensemble … view at source ↗
Figure 2
Figure 2. (Simulation experiment) Comparison of success rates [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (Simulation Experiment) Comparison of bar plots for performance metrics evaluated across 60 trials per algorithm [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Solid 3D-printed objects with a smooth finish are used [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: (Real-World Experiment) Success rates of CV-MPC [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Franka Panda robot performing the harder lateral [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: (Simulation Experiment) Performance ablation comparing Pointwise Pessimism (PWP in the figure) and Initial-State [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Snapshot of simulation ablation for different obser [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 22 canonical work pages

  1. [1]

    Human-in-the-loop imitation learning using remote tele- operation,

    A. Mandlekar, D. Xu, R. Mart ´ın-Mart´ın, Y . Zhu, L. Fei-Fei, and S. Savarese, “Human-in-the-loop imitation learning using remote tele- operation,” arXiv preprint arXiv:2012.06733 , 2020

    arXiv 2012

  2. [2]

    Roboturk: A crowdsourcing platform for robotic skill learning through imitation,

    A. Mandlekar, Y . Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay et al. , “Roboturk: A crowdsourcing platform for robotic skill learning through imitation,” in Conference on Robot Learning . PMLR, 2018, pp. 879–893

    2018

  3. [3]

    A reduction of imitation learning and structured prediction to no-regret online learning,

    S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635

    2011

  4. [4]

    Aloha unleashed: A simple recipe for robot dexterity,

    T. Z. Zhao, J. Tompson, D. Driess, P. Florence, S. K. S. Ghasemipour, C. Finn, and A. Wahid, “Aloha unleashed: A simple recipe for robot dexterity,” in 8th Annual Conference on Robot Learning

    work page

  5. [5]

    A shared-control teleoperation architecture for nonprehensile object transportation,

    M. Selvaggio, J. Cacace, C. Pacchierotti, F. Ruggiero, and P. R. Gior- dano, “A shared-control teleoperation architecture for nonprehensile object transportation,” IEEE Transactions on Robotics , 2022

    work page 2022

  6. [6]

    Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation,

    M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots, “Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation,” in Conference on Robot Learning . PMLR, 2022, pp. 750–759

    2022

  7. [7]

    Sampling-based MPC using a GPU-parallelizable physics simulator as dynamic model: an open source implementation with isaacgym,

    C. Pezzato, C. Salmi, E. Trevisan, J. A. Mora, and C. H. Corbato, “Sampling-based MPC using a GPU-parallelizable physics simulator as dynamic model: an open source implementation with isaacgym,” in Embracing Contacts - Workshop at ICRA 2023 , 2023. [Online]. Available: https://openreview.net/forum?id=fvfZKL1hCx

    work page 2023

  8. [8]

    Plan online, learn offline: Efficient learning and exploration via model-based control,

    K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch, “Plan online, learn offline: Efficient learning and exploration via model-based control,” arXiv preprint arXiv:1811.01848 , 2018

    arXiv 2018

Show all 43 references
  1. [9]

    Infinite-horizon model predictive control for periodic tasks with contacts,

    T. Erez, Y . Tassa, and E. Todorov, “Infinite-horizon model predictive control for periodic tasks with contacts,” 2012

    2012

  2. [10]

    Blending mpc & value function approximation for efficient reinforcement learning,

    M. Bhardwaj, S. Choudhury, and B. Boots, “Blending mpc & value function approximation for efficient reinforcement learning,” arXiv preprint arXiv:2012.05909, 2020

    2012 arXiv

  3. [11]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  4. [12]

    Conservative q- learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q- learning for offline reinforcement learning,” Advances in Neural In- formation Processing Systems , vol. 33, pp. 1179–1191, 2020

    2020

  5. [13]

    Autonomous helicopter aero- batics through apprenticeship learning,

    P. Abbeel, A. Coates, and A. Y . Ng, “Autonomous helicopter aero- batics through apprenticeship learning,” The International Journal of Robotics Research, vol. 29, no. 13, pp. 1608–1639, 2010

    2010

  6. [14]

    Aggressive driving with model predictive path integral control,

    G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” in 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016, pp. 1433–1440

    2016

  7. [15]

    Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,

    J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim, “Dynamic locomotion in the mit cheetah 3 through convex model-predictive control,” in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2018, pp. 1–9

    2018

  8. [16]

    Keep it upright: Model predictive control for nonprehensile object transportation with obstacle avoidance on a mobile manipulator,

    A. Heins and A. P. Schoellig, “Keep it upright: Model predictive control for nonprehensile object transportation with obstacle avoidance on a mobile manipulator,” IEEE Robotics and Automation Letters , 2023

    2023

  9. [17]

    Topp-mpc- based dual-arm dynamic collaborative manipulation for multi-object nonprehensile transportation,

    C. Zhou, M. Lei, L. Zhao, Z. Wang, and Y . Zheng, “Topp-mpc- based dual-arm dynamic collaborative manipulation for multi-object nonprehensile transportation,” in 2022 International Conference on Robotics and Automation (ICRA) , 2022

    2022

  10. [18]

    Mahalo: Unifying offline re- inforcement learning and imitation learning from observations,

    A. Li, B. Boots, and C.-A. Cheng, “Mahalo: Unifying offline re- inforcement learning and imitation learning from observations,” in International Conference on Machine Learning . PMLR, 2023, pp. 19 360–19 384

    2023

  11. [19]

    Reinforcement learning: An introduction,

    R. S. Sutton, “Reinforcement learning: An introduction,” A Bradford Book, 2018

    2018

  12. [20]

    Neuro-dynamic programming,

    D. Bertsekas, “Neuro-dynamic programming,” Athena Scientific, 1996

    1996

  13. [21]

    Morel: Model-based offline reinforcement learning,

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “Morel: Model-based offline reinforcement learning,” Advances in neural in- formation processing systems , vol. 33, pp. 21 810–21 823, 2020

    2020

  14. [22]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble,

    G. An, S. Moon, J.-H. Kim, and H. O. Song, “Uncertainty-based offline reinforcement learning with diversified q-ensemble,” Advances in neural information processing systems , vol. 34, pp. 7436–7447, 2021

    2021

  15. [23]

    Adversarially trained actor critic for offline reinforcement learning,

    C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal, “Adversarially trained actor critic for offline reinforcement learning,” in International Conference on Machine Learning . PMLR, 2022, pp. 3852–3878

    2022

  16. [24]

    Bellman- consistent pessimism for offline reinforcement learning,

    T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal, “Bellman- consistent pessimism for offline reinforcement learning,” Advances in neural information processing systems , vol. 34, pp. 6683–6694, 2021

    2021

  17. [25]

    Learning off-policy with online planning,

    H. Sikchi, W. Zhou, and D. Held, “Learning off-policy with online planning,” in Conference on Robot Learning. PMLR, 2022, pp. 1622– 1633

    2022

  18. [26]

    Model-based offline planning,

    A. Argenson and G. Dulac-Arnold, “Model-based offline planning,” arXiv preprint arXiv:2008.05556 , 2020

    2008 arXiv

  19. [27]

    An integrated system for real-time model predictive control of hu- manoid robots,

    T. Erez, K. Lowrey, Y . Tassa, V . Kumar, S. Kolev, and E. Todorov, “An integrated system for real-time model predictive control of hu- manoid robots,” in 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids) , 2013, pp. 292–299

    2013

  20. [28]

    A unified mpc framework for whole-body dynamic locomotion and manipula- tion,

    J.-P. Sleiman, F. Farshidian, M. V . Minniti, and M. Hutter, “A unified mpc framework for whole-body dynamic locomotion and manipula- tion,” IEEE Robotics and Automation Letters , vol. 6, no. 3, pp. 4688– 4695, 2021

    2021

  21. [29]

    An on- line learning approach to model predictive control,

    N. Wagener, C.-A. Cheng, J. Sacks, and B. Boots, “An on- line learning approach to model predictive control,” arXiv preprint arXiv:1902.08967, 2019

    1902 arXiv

  22. [30]

    Vp-sto: Via-point-based stochastic trajectory optimization for reactive robot behavior,

    J. Jankowski, L. Bruderm ¨uller, N. Hawes, and S. Calinon, “Vp-sto: Via-point-based stochastic trajectory optimization for reactive robot behavior,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 10 125–10 131

    2023

  23. [31]

    Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,” Advances in neural information processing systems , vol. 31, 2018

    2018

  24. [32]

    Information theoretic mpc for model-based reinforcement learning,

    G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou, “Information theoretic mpc for model-based reinforcement learning,” in 2017 IEEE international conference on robotics and automation (ICRA) . IEEE, 2017, pp. 1714–1721

    2017

  25. [33]

    Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning,

    A. S. Morgan, D. Nandha, G. Chalvatzaki, C. D’Eramo, A. M. Dollar, and J. Peters, “Model predictive actor-critic: Accelerating robot skill acquisition with deep reinforcement learning,” in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 6672–6678

    2021

  26. [34]

    Addressing function approxi- mation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- mation error in actor-critic methods,” in International conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

  27. [35]

    Critic regularized regression,

    Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess et al. , “Critic regularized regression,” Advances in Neural Information Processing Systems, vol. 33, pp. 7768–7778, 2020

    2020

  28. [36]

    Offline reinforcement learning with implicit q-learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” arXiv preprint arXiv:2110.06169 , 2021

    2021 arXiv

  29. [37]

    Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size,

    A. Nikulin, V . Kurenkov, D. Tarasov, D. Akimov, and S. Kolesnikov, “Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size,” arXiv preprint arXiv:2211.11092 , 2022

    2022 arXiv

  30. [38]

    Behavioral cloning from obser- vation,

    F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from obser- vation,” arXiv preprint arXiv:1805.01954 , 2018

    2018 arXiv

  31. [39]

    Ridm: Reinforced inverse dynamics modeling for learning from a single observed demonstration,

    B. S. Pavse, F. Torabi, J. Hanna, G. Warnell, and P. Stone, “Ridm: Reinforced inverse dynamics modeling for learning from a single observed demonstration,” IEEE Robotics and Automation Letters , vol. 5, no. 4, pp. 6262–6269, 2020

    2020

  32. [40]

    Provably efficient imitation learning from observation alone,

    W. Sun, A. Vemula, B. Boots, and D. Bagnell, “Provably efficient imitation learning from observation alone,” in International conference on machine learning . PMLR, 2019, pp. 6036–6045

    2019

  33. [41]

    Generative adversarial imitation from observation,

    F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,” arXiv preprint arXiv:1807.06158 , 2018

    2018 arXiv

  34. [42]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” Advances in neural information processing systems , vol. 29, 2016

    2016

  35. [43]

    Inter- nal model from observations for reward shaping,

    D. Kimura, S. Chaudhury, R. Tachibana, and S. Dasgupta, “Inter- nal model from observations for reward shaping,” arXiv preprint arXiv:1806.01267, 2018

    2018 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.