Pith. sign in

REVIEW 4 major objections 6 minor 71 references

A robot can use its whole arm for contact-rich tasks by planning with vision, distributed touch, and kinematics in one hybrid model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:16 UTC pith:Q57BITSS

load-bearing objection Solid systems paper: contact-centric hybrid MPC with Jacobian-shaped sampling is a real, usable package for whole-arm multi-contact, even if real-world claims rest on a soft baseline and sparse normal-only skin. the 4 major comments →

arxiv 2607.09218 v2 pith:Q57BITSS submitted 2026-07-10 cs.RO cs.AI

TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation

Rishabh Madan , Angchen Xie , Samantha Saak , Andres Blanco , Dohyeok Lee , Sarah Grace Brown , Yunting Yan , Mark Zolotas
show 2 more authors
Jose Barreiros Tapomayukh Bhattacharjee
This is my paper
classification cs.RO cs.AI
keywords whole-arm manipulationdistributed tactile sensingsampling-based MPCcontact Jacobianshybrid predictive modelsvisuotactile controlforce regulationreceding-horizon control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Whole-arm manipulation asks a robot to make, slide, and break contact along multiple links while still finishing a job and keeping forces safe. Pure learning often fails here because multi-link contact is rare in data and rollouts can become physically inconsistent; pure analytical control often misses partially observed, multimodal contact. This paper claims that a receding-horizon controller called TACTIC can close that gap by building a contact-centric state from RGB-D, distributed tactile readings, and a compact proximity map, then rolling out future contact and force with a hybrid model that couples learned latent dynamics to analytical kinematics through contact Jacobians. Sampling-based planning is steered with those Jacobians so candidate motions explore force-modulating directions, and scores trade task progress against whole-arm force limits. In simulation and on a real arm with tactile skin, the approach improves success and reduces unsafe force events versus pure learned and pure analytical alternatives on maze navigation, manikin rollover, and limb repositioning.

Core claim

The paper establishes that whole-arm multi-contact control becomes more successful and safer when planning is contact-centric: fuse vision, distributed tactile sensing, and proximity into a shared state; predict with a hybrid model that pairs learned latent dynamics with analytical kinematics via contact Jacobians; and sample actions with Jacobian projections that bias exploration toward force-modulating directions while scoring trajectories on predicted proximity and interaction forces.

What carries the argument

TACTIC’s contact-centric hybrid predictive model plus contact-aware action sampling: a learned action-conditioned latent dynamics model is coupled to analytical kinematics through contact Jacobians so the planner can roll out future contact configurations and forces, while force-weighted normal Jacobians reshape MPPI samples toward force-regulating directions and safety cones.

Load-bearing premise

The method treats active contact points, surface normals, and force-weighted contact Jacobians as locally valid over the short planning horizon, and it only senses normal force at discrete instrumented patches on the arm.

What would settle it

On the same multi-contact tasks, disable contact-Jacobian shaping and hybrid force scoring while keeping the rest of the controller fixed; if success and force-violation rates then match or beat full TACTIC, the central claim that those contact-centric pieces are necessary would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TACTIC is a receding-horizon sampling-based MPC controller for whole-arm multi-contact manipulation. It fuses RGB-D, distributed tactile forces, and a 2D proximity mask into a contact-centric latent state; biases MPPI exploration with force-weighted normal contact Jacobians and a safety-cone QP; and scores hybrid rollouts that couple analytical kinematics (via contact Jacobians and a spring force model) with a learned JEPA-style latent dynamics model. Simulation ablations on a 3D maze and related tasks attribute gains to representation, contact-aware sampling, and hybrid prediction (Tables II–V), and real-robot experiments on a Kinova Gen3 with 22 FlexiForce taxels report higher success than Diffusion Policy on manikin side-rollover and limb repositioning, with often fewer force-threshold violations than matched expert teleoperation (Table VI).

Significance. If the claims hold, the paper offers a concrete, deployable recipe for force-aware whole-arm control that is more than end-effector-centric imitation or pure world-model RL: contact Jacobians are used not only for scoring but for shaping the sampler, and hybrid rollouts reduce physical inconsistency under sparse multi-contact data. The open hardware (exoskeleton teleop and WiFi piezoresistive skin) and the isolation of three design choices in simulation are genuine contributions for assistive and cluttered-contact robotics. The work is significant for pHRI and caregiving-style tasks where multi-link contact is necessary rather than incidental, provided real-world force regulation generalizes beyond the instrumented, normal-dominated regime used here.

major comments (4)
  1. [§VI, Table VI] Table VI is load-bearing for the real multi-contact claim, but the only autonomous baseline is Diffusion Policy. DP’s Maze FV=0 is explained as top-entry avoidance of whole-arm contact, so the comparison does not stress the same multi-contact regime TACTIC targets. Model-based methods that already trail TACTIC in simulation (DreamerV3, TD-MPC2; Table V) are omitted on the robot with only a brief justification. Either add at least one stronger real baseline (or a contact-forcing DP/IL variant), or reframe Table VI as a demonstration with explicit limits on what the SR/FV gaps establish.
  2. [§IV-B, Eqs. (2)–(5); Appendix C, E, G] Contact-aware sampling (force-weighted normal Jacobian projectors G_force/G_null and the safety QP in Eqs. 2–5) and the analytical force costs (Appendix E, Eqs. 9–15) treat active taxels, outward normals, and J^n as locally constant over H=8. Appendix G states discrete normal-only coverage and uninstrumented regions; manikin rollover and limb lift are expected to involve sliding, off-taxel contact, and shear. The manuscript needs either (i) empirical checks that active contacts remain taxel-localized and normals stable over the horizon on the real tasks, or (ii) a sensitivity/failure analysis when those assumptions break, so that reported FV reductions are not over-attributed to Jacobian shaping under ideal sensing.
  3. [Appendix E, Eqs. (9)–(15); Appendix C Table VIII] The hybrid force objective mixes a learned binary contact decoder with a linear spring model whose stiffness k_spr is fit offline (Appendix E). F_safe, f_min, F_sat, and the shaping scales (σ_force, σ_null, ε) are free parameters that directly enter both sampling and c_force. There is no reported sensitivity of SR/FV to k_spr or to mis-specified normals/thresholds on the real manikin tasks. Without that, it is hard to separate algorithmic gains from careful threshold tuning on the same sensor suite used at train and test time.
  4. [§VI Table VI; §V Tables II–V] Real success rates are reported over small N (5 maze trials; 20 rollover/limb trials) with large FV standard deviations (e.g., Maze TACTIC FV 70.3±29.4). Simulation FV variances are similarly large (Table II–V). The paper should report confidence intervals or binomial tests for SR differences versus DP/expert, and clarify whether FV counts are per-trial totals or rates, so that “fewer violations than expert teleop” is statistically interpretable rather than descriptive.
minor comments (6)
  1. [Title / §I] Title and early headings in the source show broken spacing (“TACTIC:T actilea nd VisionC onditioned…”); clean for camera-ready.
  2. [Table I; §IV-B] Glossary Table I is useful but dense; define K_t vs |K_t| and F_sat vs F_safe once in the main text near first use.
  3. [Appendix B1] Proximity mask uses d_max=0.05 m in Appendix B; confirm this is intentional for “impending contact” and state units consistently with f_min=2 N.
  4. [§IV-C5; Appendix D] Two-fidelity draft safeguard (τ_draft=0.1, T_full=4) is important for real-time claims; a short runtime breakdown of how often fallback triggers on the real tasks would strengthen §VI.
  5. [Abstract] Website URL appears both with and without https in abstract/intro; standardize.
  6. [§II] Related work on whole-arm tactile skins and visuotactile MPC is appropriate; a short pointer to how TACTIC differs from Killpack/Kemp-style force-regulating MPC in clutter would help readers place C2–C3.

Circularity Check

0 steps flagged

No significant circularity: TACTIC is an empirical systems/control paper whose performance claims rest on external baselines and ablations, not on predictions forced by definition or self-citation chains.

full rationale

The paper proposes a receding-horizon controller (contact-centric representation, Jacobian-shaped MPPI sampling, hybrid latent+kinematic rollouts, force/proximity costs) and evaluates it with success rate, time-to-completion, and force-violation metrics against independent methods (DreamerV3, TD-MPC2, Diffusion Policy, STORM ablations, V-JEPA2/DINO-WM hybrids) and component ablations (no tactile/mask, no contact-aware sampling, latent-only vs kinematics-only). None of the load-bearing claims reduce by construction to their inputs: contact Jacobians and spring force estimates are analytical design choices scored inside MPC, not fitted quantities re-labeled as independent predictions of the same data; the latent dynamics are trained offline and judged by task outcomes under distribution shift, not by recovering the training loss; self-citations (PrioriTouch, CushSense, RCareWorld) supply context or datasets, not uniqueness theorems that force the method. Offline hyperparameter tuning (k_spr, F_safe, shaping sigmas) and IQL value training on demonstration progress variables are standard engineering practice and do not make reported SR/FV gains tautological. No self-definitional loop, fitted-input-as-prediction, or load-bearing self-citation chain is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central performance claim rests on standard robot kinematics and MPC practice plus several modeling choices that are not free theorems: taxel-localized normal contacts, short-horizon constant normals, a linear spring force model, offline-tuned safety and sampling scales, and the assumption that offline expert/random-play data plus a latent JEPA-style predictor generalize enough for receding-horizon multi-contact control. No new physical entities are postulated; the invented pieces are engineering constructs (proximity mask, hybrid predictor, contact-aware sampler).

free parameters (5)
  • F_safe / f_min / F_sat force thresholds
    Safety and active-contact thresholds that define violations, contact sets, and sampling weights; chosen for hardware and tasks rather than derived.
  • k_spr spring stiffness for analytical force estimates
    Linear spring maps predicted normal compression to force for c_active/c_pred; fit offline (Appendix E).
  • Contact-shaping scales sigma_force, sigma_null, sigma_min, ridge epsilon
    Hand/sim-selected scales that bias MPPI into contact-normal vs nullspace directions (Appendix C Table VIII).
  • MPC/MPPI hyperparameters H, N, M, L, Delta t, cost weights, lambda_V
    Horizon, particle counts, two-fidelity top-M, history length, and cost/value weights control planner behavior and are tuned, not derived.
  • Proximity mask parameters d_max, taxel blob radius r
    Visual proximity clipping distance and taxel rendering radius set the contact-centric image representation (Appendix B).
axioms (5)
  • domain assumption Robot kinematics and contact Jacobians accurately map joint velocities to normal motion at taxel centers for active contacts.
    Used throughout analytical branch F_k, sampling projectors G_force/G_null, and spring force updates (Sec. IV-B/C).
  • domain assumption Active contact normals and contact set remain approximately constant over the short MPC horizon.
    Explicitly assumed for contact-conditioned shaping and safety QP (Sec. IV-B, Appendix C).
  • domain assumption Normal-only discrete taxel forces plus a proximity mask sufficiently observe task-relevant multi-link contact under occlusion.
    Core sensing premise; limitations note uninstrumented regions and missing shear (Appendix G).
  • domain assumption Offline expert and random-play trajectories are diverse enough for latent dynamics and IQL value learning to support online multi-contact planning.
    All learning components are trained offline with no online RL interaction (Sec. V Data and Training Protocol).
  • ad hoc to paper Sampling-based receding-horizon MPC with hybrid rollouts can trade task progress against force regulation without formal safety certificates.
    Design principle of the controller; discussion admits no formal guarantees and relies on low-level compliance plus planning costs (Sec. VII).
invented entities (3)
  • Contact-centric observation o_cc including fused proximity mask M_t no independent evidence
    purpose: Ground impending and active whole-arm contact in an image-aligned representation for encoding and planning.
    Constructed from YOLO segmentation, kd-tree distances, and taxel projections; engineering representation, not a physical discovery.
  • Contact-aware action sampling via force-weighted normal Jacobian projectors and safety-cone QP no independent evidence
    purpose: Bias MPPI samples toward force-modulating directions and away from high-force penetration.
    Algorithmic construct built from contact Jacobians; validated by ablation, not independently measured outside the controller.
  • Hybrid predictive model F = {F_k, F_l} with two-fidelity latent rollouts no independent evidence
    purpose: Combine analytical kinematics consistency with learned contact/force evolution for trajectory scoring.
    System-level model composition; performance evidence is internal to the paper's experiments.

pith-pipeline@v1.1.0-grok45 · 28074 in / 3789 out tokens · 29476 ms · 2026-07-14T15:16:17.526249+00:00 · methodology

0 comments
read the original abstract

Whole-arm manipulation involves direct contact with the environment while the robot completes a task by distributing contact across multiple links as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and contact forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many multi-link contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm manipulation. TACTIC uses a contact-centric hybrid predictive model that combines RGB-D, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurations and interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling: contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. TACTIC consistently outperforms other methods. We further demonstrate real-world performance on a robot with distributed tactile sensing across three whole-arm manipulation tasks that require multi-contact trajectories: turning over and repositioning a manikin, and goal-reaching in a 3D dynamic maze. Website: https://emprise.cs.cornell.edu/tactic

Figures

Figures reproduced from arXiv: 2607.09218 by Andres Blanco, Angchen Xie, Dohyeok Lee, Jose Barreiros, Mark Zolotas, Rishabh Madan, Samantha Saak, Sarah Grace Brown, Tapomayukh Bhattacharjee, Yunting Yan.

Figure 1
Figure 1. Figure 1: TACTIC enables robots to perform contact-rich manipulation by utilizing the entire arm surface. While end-effector-centric control is often infeasible and unsafe under high payloads, TACTIC fuses multimodal observations such as RGB-D, proximity masks, and distributed tactile sensing into a contact-centric representation to enable whole-arm manipulation. We demonstrate its effectiveness on manipulation task… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TACTIC. A. Contact-Centric Representation encodes multimodal history into a latent state zt. B. Contact-Aware Action Sampling utilizes tactile-derived contact Jacobians (Jk,t) to bias MPPI exploration toward force-modulating directions. C. Contact￾Centric Hybrid Predictive Model evaluates candidate sequences by combining an analytical kinematics branch for physical consistency with a learned la… view at source ↗
Figure 3
Figure 3. Figure 3: Simulation task suite. We evaluate TACTIC across four simulation environments: the 3D Maze Environment tests whole-arm navigation through a maze, analogous to navigation in a cluttered environment; Bed Bathing tests task execution of wiping a person in bed with incidental whole-arm contact; Tabletop Reach tests obstacle-aware reaching across a table; and Granular Manipulation tests contact-driven rearrange… view at source ↗
Figure 4
Figure 4. Figure 4: Real-world task suite. We evaluate across three real-world environments. The 3D Maze Environment tests whole-arm navigation through static and dynamic obstacles, Side Rollover tests contact-rich manipulation to rotate a person while regulating interaction forces, and Limb Repositioning tests multi-contact support during lifting and repositioning of the legs to a target configuration. Whole-arm contact is h… view at source ↗
Figure 5
Figure 5. Figure 5: Real-world setup. Kinova Gen3 7-DoF arm with 22 dis￾tributed tactile sensors, RGB-D perception, and exoskeleton-based teleoperation for data collection. dynamic cluttered maze that tests contact-rich navigation in￾volving highly constrained multi-contact trajectories, and (ii) contact-intensive manipulation tasks with a life-size manikin (∼46 lbs, vinyl plastic) of the type used for nurse training on the t… view at source ↗
Figure 6
Figure 6. Figure 6: Cosine similarity for RGB, Depth, and Proximity Mask. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 12 linked inside Pith

  1. [1]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  2. [2]

    A careful examination of large behavior models for multitask dexterous manipulation

    Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    V-jepa 2: Self-supervised video models enable un- derstanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Gar- rido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable un- derstanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. InTowards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023, 2023

    2023

  5. [5]

    Whole-arm tactile sensing for beneficial and acceptable contact dur- ing robotic assistance

    Phillip M Grice, Marc D Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C Kemp. Whole-arm tactile sensing for beneficial and acceptable contact dur- ing robotic assistance. In2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), pages 1–8. IEEE, 2013

    2013

  6. [6]

    Reaching in clutter with whole-arm tactile sensing.The International Journal of Robotics Research, 32(4):458–482, 2013

    Advait Jain, Marc D Killpack, Aaron Edsinger, and Charles C Kemp. Reaching in clutter with whole-arm tactile sensing.The International Journal of Robotics Research, 32(4):458–482, 2013

    2013

  7. [7]

    Killpack and Charles C

    Marc D. Killpack and Charles C. Kemp. Fast reaching in clutter while regulating forces using model predictive control. In2013 13th IEEE-RAS International Confer- ence on Humanoid Robots (Humanoids), pages 146–153, 2013

    2013

  8. [8]

    Discover our gen3 robotic arm

    Kinova Robotics. Discover our gen3 robotic arm. https: //www.kinovarobotics.com/product/gen3-robots. Ac- cessed: 2026-05-21

    2026

  9. [9]

    Cushsense: Soft, stretchable, and comfortable tactile-sensing skin for physical human-robot interaction

    Boxin Xu, Luoyan Zhong, Grace Zhang, Xiaoyu Liang, Diego Virtue, Rishabh Madan, and Tapomayukh Bhat- tacharjee. Cushsense: Soft, stretchable, and comfortable tactile-sensing skin for physical human-robot interaction. arXiv preprint arXiv:2405.03155, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Robotsweater: Scal- able, generalizable, and customizable machine-knitted tactile skins for robots, 2023

    Zilin Si, Tianhong Catherine Yu, Katrene Morozov, James McCann, and Wenzhen Yuan. Robotsweater: Scal- able, generalizable, and customizable machine-knitted tactile skins for robots, 2023

    2023

  11. [11]

    Tsui, and Alex Alspach

    Aimee Goncalves, Naveen Kuppuswamy, Andrew Beaulieu, Avinash Uttamchandani, Katherine M. Tsui, and Alex Alspach. Punyo-1: Soft tactile-sensing upper- body robot for large object manipulation and physical human interaction. In2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), pages 844–851, 2022

    2022

  12. [12]

    A warm hug from a robot: A dual-mode e-skin with programming compliance.Review of Scientific Instruments, 93(11), 2022

    Zhe Liu, Diansheng Chen, Junlin Ma, Zhenyang Shen, Tianhao Wu, Zining Jia, and Yongkang Jiang. A warm hug from a robot: A dual-mode e-skin with programming compliance.Review of Scientific Instruments, 93(11), 2022

    2022

  13. [13]

    Geometry-based two-contact inverse kinematic solution for whole arm manipulation

    Pascal Gliesche, Christian Kowalski, Max Pfingsthorn, and Andreas Hein. Geometry-based two-contact inverse kinematic solution for whole arm manipulation. In2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8269–8274. IEEE, 2021

    2021

  14. [14]

    Learn- ing contact-rich whole-body manipulation with example- guided reinforcement learning.Science Robotics, 10(105):eads6790, 2025

    Jose A Barreiros, Aykut ¨Ozg¨un ¨Onol, Mengchao Zhang, Sam Creasey, Aimee Goncalves, Andrew Beaulieu, Aditya Bhat, Kate M Tsui, and Alex Alspach. Learn- ing contact-rich whole-body manipulation with example- guided reinforcement learning.Science Robotics, 10(105):eads6790, 2025

    2025

  15. [15]

    Priori- touch: Adapting to user contact preferences for whole- arm physical human-robot interaction.arXiv preprint arXiv:2509.18447, 2025

    Rishabh Madan, Jiawei Lin, Mahika Goel, Angchen Xie, Xiaoyu Liang, Marcus Lee, Justin Guo, Pranav N Thakkar, Rohan Banerjee, Jose Barreiros, et al. Priori- touch: Adapting to user contact preferences for whole- arm physical human-robot interaction.arXiv preprint arXiv:2509.18447, 2025

    arXiv 2025

  16. [16]

    The six hug commandments: Design and evaluation of a human- sized hugging robot with visual and haptic perception

    Alexis E Block, Sammy Christen, Roger Gassert, Otmar Hilliges, and Katherine J Kuchenbecker. The six hug commandments: Design and evaluation of a human- sized hugging robot with visual and haptic perception. InProceedings of the 2021 ACM/IEEE international conference on human-robot interaction, pages 380–388, 2021

    2021

  17. [17]

    Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg

    Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks, 2019

    2019

  18. [18]

    More than a feeling: Learning to grasp and regrasp using vision and touch

    Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H Adelson, and Sergey Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300–3307, 2018

    2018

  19. [19]

    Visuo-tactile transformers for manipulation

    Yizhou Chen, Andrea Sipos, Mark Van der Merwe, and Nima Fazeli. Visuo-tactile transformers for manipulation. arXiv preprint arXiv:2210.00121, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    Manipulation by feel: Touch-based control with deep predictive models

    Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, and Sergey Levine. Manipulation by feel: Touch-based control with deep predictive models. In2019 International Confer- ence on Robotics and Automation (ICRA), pages 818–

  21. [21]

    Letac-mpc: Learning model predictive control for tactile-reactive grasping.IEEE Transactions on Robotics, 2024

    Zhengtong Xu and Yu She. Letac-mpc: Learning model predictive control for tactile-reactive grasping.IEEE Transactions on Robotics, 2024

    2024

  22. [22]

    A contact model based on denoising diffusion to learn variable impedance control for contact-rich ma- nipulation

    Masashi Okada, Mayumi Komatsu, and Tadahiro Taniguchi. A contact model based on denoising diffusion to learn variable impedance control for contact-rich ma- nipulation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7286–

  23. [23]

    Irasim: A fine-grained world model for robot manipulation

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 9834–9844, 2025

    2025

  24. [24]

    Worldplanner: Monte carlo tree search and mpc with action-conditioned visual world models

    R Khorrambakht, Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, and Ludovic Righetti. Worldplanner: Monte carlo tree search and mpc with action-conditioned visual world models. arXiv preprint arXiv:2511.03077, 2025

    arXiv 2025

  25. [25]

    Model predictive path integral control: From theory to parallel computation.Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017

    Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation.Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017

    2017

  26. [26]

    Yolov11: An overview of the key architectural enhancements.arXiv preprint arXiv:2410.17725, 2024

    Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements.arXiv preprint arXiv:2410.17725, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Analysis of approximate nearest neighbor searching with clustered point sets.arXiv preprint cs/9901013, 1999

    Songrit Maneewongvatana and David M Mount. Analysis of approximate nearest neighbor searching with clustered point sets.arXiv preprint cs/9901013, 1999

    Pith/arXiv arXiv 1999

  28. [28]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    An intriguing failing of convolutional neural networks and the coordconv solution.Advances in neural infor- mation processing systems, 31, 2018

    Rosanne Liu, Joel Lehman, Piero Molino, Felipe Pet- roski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution.Advances in neural infor- mation processing systems, 31, 2018

    2018

  30. [30]

    Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation

    Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan D Ratliff, Dieter Fox, Fabio Ramos, and Byron Boots. Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation. InConference on Robot Learning, pages 750–759. PMLR, 2022

    2022

  31. [31]

    Dino-wm: World models on pre-trained vi- sual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained vi- sual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  33. [33]

    What drives success in physical planning with joint-embedding predictive world models?arXiv preprint arXiv:2512.24497, 2025

    Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, and Yann LeCun. What drives success in physical planning with joint-embedding predictive world models?arXiv preprint arXiv:2512.24497, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Pith/arXiv arXiv 2021

  35. [35]

    Rcareworld: A human- centric simulation world for caregiving robots

    Ruolin Ye, Wenqiang Xu, Haoyuan Fu, Rajat Kumar Je- namani, Vy Nguyen, Cewu Lu, Katherine Dimitropoulou, and Tapomayukh Bhattacharjee. Rcareworld: A human- centric simulation world for caregiving robots. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 33–40. IEEE, 2022

    2022

  36. [36]

    Robohive: A unified framework for robot learning.Advances in Neural Information Processing Systems, 36:44323–44340, 2023

    Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning.Advances in Neural Information Processing Systems, 36:44323–44340, 2023

    2023

  37. [37]

    Robohive Franka environ- ment for V-JEPA2

    Lasse Sofus Møller Sørensen. Robohive Franka environ- ment for V-JEPA2. https://github.com/lassesofus/thesis,

  38. [38]

    Accessed: 2026-05-11

    2026

  39. [39]

    Adaptigraph: Material-adaptive graph-based neu- ral dynamics for robotic manipulation.arXiv preprint arXiv:2407.07889, 2024

    Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Adaptigraph: Material-adaptive graph-based neu- ral dynamics for robotic manipulation.arXiv preprint arXiv:2407.07889, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Mastering diverse control tasks through world models.Nature, 640(8059):647–653, April 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Tim- othy Lillicrap. Mastering diverse control tasks through world models.Nature, 640(8059):647–653, April 2025

    2025

  41. [41]

    Td-mpc2: Scalable, robust world models for continuous control, 2024

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024

    2024

  42. [42]

    FlexiForce load/force sensors and systems

    Tekscan, Inc. FlexiForce load/force sensors and systems. https://www.tekscan.com/products/flexiforce-sensors, 2024

    2024

  43. [43]

    Dif- fusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- fusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  44. [44]

    easy handeye: Automated, hardware- independent hand-eye calibration for ROS

    Marco Esposito. easy handeye: Automated, hardware- independent hand-eye calibration for ROS. https://github. com/IFL-CAMP/easy handeye. Accessed: 2026-07-09. TABLE OFCONTENTS A System Overview and Latency Breakdown13 1 Pipeline overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2 Hyperparameter table ...

    2026

  45. [45]

    We acquire synchronized sensor data at40Hz

    Pipeline overview At each MPC stept, the system runs the following se- quence:Sensor inputs→Mask generation→Contact- centric observation→Encoding→Rollouts (draft + kinematic + full) + decoding→Scoring→Execution. We acquire synchronized sensor data at40Hz. We then generate the proximity maskM t (segmentation + kd-tree queries + taxel projection) which take...

  46. [46]

    Parameter Value MPC horizonH8 MPC time step∆t0.025 MPPI particlesN100 Two-fidelity top-M10 Latent history lengthL3 Control modejoint-space impedance B

    Hyperparameter table TABLE VII: Hyperparameters for TACTIC. Parameter Value MPC horizonH8 MPC time step∆t0.025 MPPI particlesN100 Two-fidelity top-M10 Latent history lengthL3 Control modejoint-space impedance B. CONTACT-CENTRICREPRESENTATIONDETAILS

  47. [47]

    We use a fixed extrinsic transform between the robot base and camera, which is needed only for projecting taxels into the image

    Proximity mask construction We assume RGB and depth are spatially aligned and obtain camera intrinsicsKusing thepyrealsenseAPI. We use a fixed extrinsic transform between the robot base and camera, which is needed only for projecting taxels into the image. We use theeasy_handeye[43] package to obtain the extrinsics. For each pixel(u, v)with valid depthz(u...

  48. [48]

    6: Cosine similarity for RGB, Depth, and Proximity Mask

    Encoder implementation details Fig. 6: Cosine similarity for RGB, Depth, and Proximity Mask. Vision backbone.We apply the same DINOv3 backbone toI t,D t, andM t in a batched forward pass. Depth and mask inputs are replicated to three channels and normalized using the same preprocessing as RGB (after scaling depth to metric units). The DINO feature map is ...

  49. [49]

    For each active taxelk, we treat the contact point as the taxel center obtained from forward kinematics

    Active contact set and contact geometry We define the active contact setK t from taxels whose measured normal force exceeds thresholdf min. For each active taxelk, we treat the contact point as the taxel center obtained from forward kinematics. The corresponding translational Ja- cobianJ k,t ∈R 3×n is computed from the robot model. Each taxel has a known ...

  50. [50]

    We computeG force t using the small matrix inverse( fJntfJn⊤ t +εI) −1 with a stable factorization (e.g., Cholesky)

    Force weighting and projection operators We use the taxel force magnitudef k,t (scalar normal force) and defineρ k,t = sat(f k,t/Fsat)to reduce the influence of weak/noisy contacts while preventing a single contact from dominating via saturation. We computeG force t using the small matrix inverse( fJntfJn⊤ t +εI) −1 with a stable factorization (e.g., Chol...

  51. [51]

    Because|K viol t |is small, we solve it efficiently with an active-set method

    Safety projection QP We formK viol t ={k∈ K t |f k,t ≥F safe}and stack the corresponding rows intoJ viol t . Because|K viol t |is small, we solve it efficiently with an active-set method. All2K active sets are enumerated, the corresponding equality-constrained least- squares problems are solved in closed form, and the feasible solution with minimum object...

  52. [52]

    We trainF draft l purely using teacher forcing loss (imitate the full model)

    Two-fidelity rollouts for real-time control Draft model design and training.The draft predictorF draft l shares the same encoders and action embeddingϕ act as the full predictor, but uses a lightweight two-layer MLP. We trainF draft l purely using teacher forcing loss (imitate the full model). Selection and rescoring.At each MPC stept, we sampleN action s...

  53. [53]

    Joint limits.Let(q min,q max)be joint bounds and∆ q a soft margin

    Kinematic costc kin We decomposec kin(ˆst+i)into feasibility and task terms: ckin(ˆst+i) =λ lim clim(ˆqt+i) +λ coll ccoll(ˆqt+i) +λ ee cee(ˆet+i) +λ u cu(ut+i), (6) withλ lim = 1000.0,λ coll = 1000.0,λ ee = [15.0,10.0] (orientation, position), andλ u = 1.0. Joint limits.Let(q min,q max)be joint bounds and∆ q a soft margin. We use clim(ˆq) = nX m=1 [ˆqm−(q...

  54. [54]

    For eachk∈ Kt, we form a spring-based force estimate along the measured contact normal using the kinematic rollout ˆqt+1:t+H

    Active force costc active LetK t denote the active contact set at timet. For eachk∈ Kt, we form a spring-based force estimate along the measured contact normal using the kinematic rollout ˆqt+1:t+H. For taxel k, letJ n k,t =n ⊤ k,tJk,t be the normal Jacobian row. Given a kinematic rollout ˆqt:t+H, define∆q t+i = ˆqt+i − ˆqt+i−1 and the predicted normal di...

  55. [55]

    We first use the binary contact decoderψ contact to estimate the contact onset timeˆτ k for each taxelk

    Predicted force costc pred Instead of decoding raw forces directly from the predicted latents (which can be noisy when evaluated far into the horizon), we derive a stable force signal for scoring from the decoded binary contacts and joint positions. We first use the binary contact decoderψ contact to estimate the contact onset timeˆτ k for each taxelk. We...

  56. [56]

    Latent goal costc goal We encode the goal observation asz g =ϕ(o cc g ), whereo cc g is formed analogously too cc t (Sec. IV-A). The goal cost is cgoal(ˆzt+i,z g) =∥ ˆzt+i −z g∥2 2.(16)

  57. [57]

    Infeasible rollouts (large joint-limit violations or self- collision) are handled by increasing the corresponding penalty by a factor of1000rather than rejecting the sample

    Weights and implementation Costs are summed over the horizon with discount factorγ= 0.98. Infeasible rollouts (large joint-limit violations or self- collision) are handled by increasing the corresponding penalty by a factor of1000rather than rejecting the sample. F. IQL VALUELEARNING FORLONG-HORIZONTASKS

  58. [58]

    We obtain the values for these indicators during data collection by mounting ArUco markers on the head and limb joints

    Progress Variables and Subgoals For each long-horizon task, the dataset provides low- dimensional progress variablesξ t ∈R dg (e.g., body turn angle, limb joint angles). We obtain the values for these indicators during data collection by mounting ArUco markers on the head and limb joints. Note that these markers are not required at test time. Instead, we ...

  59. [59]

    IV-A) and com- pute a latent embeddingz t for each observation

    Latent Encoding and Goal Representation We freeze the multimodal encoders (Sec. IV-A) and com- pute a latent embeddingz t for each observation. For each subgoalξ (g) g , we also form a corresponding goal observation and embed it with the same frozen encoders to obtain the goal latentz g. The choice of subgoals depends on task difficulty. For instance, in ...

  60. [60]

    Offline Transition Dataset From the expert dataset, we construct an offline transition set D={(z t,u t,z t+1,z g, rt, dt)},(18) whereu t is the executed action,d t ∈ {0,1}indicates episode termination, and the reward is defined as the negative of the cost used for scoring task state (Sec. IV-D): rt =−ℓ t.(19) ℓt includes a progress/subgoal tracking term b...

  61. [61]

    Q-learning objective.The TD target is yt =r t +γ(1−d t)V ¯θ(xt+1),(22) and we minimize LQ =E D h (Qϕ1(xt,u t)−y t)2 + (Qϕ2(xt,u t)−y t)2 i

    Goal-Conditioned IQL Critic We implement goal-conditioning by concatenating the state and goal latents (and optionally their difference): xt = [zt,z g,(z g −z t)].(21) We learn double action-value functionsQ ϕ1(xt,u t), Qϕ2(xt,u t)and a value functionV θ(xt), with a slowly- updated target value networkV ¯θ. Q-learning objective.The TD target is yt =r t +γ...

  62. [62]

    We then augment the rollout cost as Jaug(ut:t+H−1) =J(u t:t+H−1)−λ V Vθ(ˆzt+H ,z g),(25) whereV θ is the learned IQL value function

    Using the Value for Terminal Shaping in MPPI At test time, MPPI rolls out the learned latent predictorF l to obtain the horizon-end latent ˆzt+H for each sampled action sequence. We then augment the rollout cost as Jaug(ut:t+H−1) =J(u t:t+H−1)−λ V Vθ(ˆzt+H ,z g),(25) whereV θ is the learned IQL value function. SinceV θ estimates reward-to-go and MPPI mini...

  63. [63]

    Each pad measures normal force over a50.8 mm×50.8 mmsensing area and outputs a resistance change proportional to applied load

    Hardware overview We mount 22 Tekscan FlexiForce A502 force-sensing resis- tors (FSRs) on the robot to measure distributed normal contact forces along the arm. Each pad measures normal force over a50.8 mm×50.8 mmsensing area and outputs a resistance change proportional to applied load. Sensors are grouped into four physical mounts: the two wrist joints (5...

  64. [64]

    A tare is accepted only when the window variance is below a stability threshold

    Sensor readings and preprocessing Taring.Before interaction, we tare each sensor by estimat- ing its no-contact offset from a recent window of raw readings. A tare is accepted only when the window variance is below a stability threshold. Filtering.We apply a median filter followed by a low-pass filter to the tared signal. The resulting tactile vectorf t ∈...

  65. [65]

    We obtain taxel positions and outward normals in the link frame from manual measurements (vernier caliper) and CAD alignment, and include these extrinsics in the robot model

    Taxel pose, normals, and contact Jacobians Taxel extrinsics.For each taxelk, we define a taxel frame {Tk}attached to the sensing pad and store its fixed transform with respect to the corresponding link frame{L}, LTTk. We obtain taxel positions and outward normals in the link frame from manual measurements (vernier caliper) and CAD alignment, and include t...

  66. [66]

    We repeatedly apply normal loads over a range of magnitudes and fit a linear model that maps raw readings to force in Newtons using least-squares regression

    Force calibration We calibrate the skin sensor using an ATI Nano25 force/- torque sensor. We repeatedly apply normal loads over a range of magnitudes and fit a linear model that maps raw readings to force in Newtons using least-squares regression

  67. [67]

    The driver includes a watchdog for common failure modes (e.g., missing packets, low battery, disconnected boards, inactive channels)

    Driver and data integrity checks We implement askin_driverthat receives UDP packets from the WiFi boards and exposes ROS services and topics for taring and reading tactile values. The driver includes a watchdog for common failure modes (e.g., missing packets, low battery, disconnected boards, inactive channels). The driver performs the filtering discussed...

  68. [68]

    Limitations Coverage.The tactile skin provides discrete coverage over a subset of the arm surface; contact can occur in uninstrumented regions.Normal-only sensing.The sensors measure normal force only. This ignores tangential forces and frictional effects (e.g., shear during sliding), which can be informative for both scoring and sampling in contact-rich ...

  69. [69]

    Exoskeleton teleoperation interface We collect human demonstration data using a custom ex- oskeleton teleoperation interface that enables mirrored whole- arm control of a Kinova Gen3 (7-DoF). Mirroring human arm motion is important for capturing whole-arm contact strategies relevant to caregiving, including distributing con- tact across links and coordina...

  70. [70]

    RGB and depth images are captured with an Intel RealSense D455 at 60 Hz and time-synchronized to the 40 Hz logging stream

    Offline dataset summary During data acquisition, RGB-D, tactile, and proprioceptive observations are recorded synchronously at 40 Hz. RGB and depth images are captured with an Intel RealSense D455 at 60 Hz and time-synchronized to the 40 Hz logging stream. In the 3D Maze environment, we collected 125 expert and 125 random-play episodes (each∼30 s). For Si...

  71. [71]

    To make the model hybrid, we do FK to compute end-effector samples and add a cost for end-effector goal reaching

    V-JEPA2 and DINO-WM experiment details Finetuning V-JEPA2 and Hybrid Model Experiment.We load the pre-trained encoders and perform action-conditioned finetuning of the predictor using the tabletop reach dataset. To make the model hybrid, we do FK to compute end-effector samples and add a cost for end-effector goal reaching. This cost forces the planner to...