Pith. sign in

REVIEW 3 major objections 6 minor 8 cited by

VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VLA-Touch claims that adding tactile feedback at both the planning and control levels makes a vision-language-action model plan faster and manipulate more precisely, without fine-tuning the base model.

desk verdict The integration is real and the real-robot results are worth a look, but the paper's central 'without fine-tuning' claim is contradicted by its own appendix, so the framing needs honest revision. read the letter →

arxiv 2507.17294 v2 pith:UEJ26R4I submitted 2025-07-23 cs.RO cs.LG

Jianxin Bi , Kevin Yuchen Ma , Ce Hao , Mike Zheng Shou , Harold Soh This is my paper · ORCID
classification cs.ROcs.LG
keywords tactileperceptionvision-language-actionmodelsinterpolantdiffusioncontact-richmanipulationtactile-languagemodelrobotmultimodalfeedback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VLA-Touch claims that a vision-language-action (VLA) robot policy can be made much better at contact-rich tasks by feeding it tactile information at two separate levels, without altering the VLA's own weights. At the planning level, a pre-trained tactile-language model converts touch sensor images into short textual descriptions of object hardness and roughness, which a vision-language task planner uses to make decisions that vision alone cannot. At the control level, a small diffusion-based interpolant controller refines the VLA's generated action chunks using low-dimensional tactile force estimates before execution. Real-robot experiments on cup placement, sponge wiping, and mango peeling report up to 40% better planning success, up to 35% higher manipulation success over vision-only baselines, and up to 35% better task success than using tactile feedback at only one level.

What carries the argument

The load-bearing piece is a two-channel interface around a frozen VLA. Upstream, a pretrained tactile-language model converts six GelSight tactile frames into scalar hardness and roughness descriptions that are returned to a vision-language planner in a closed perception-planning loop. Downstream, an interpolant diffusion controller diffuses from the VLA action chunk as an informative source distribution toward expert actions, conditioned on visual embeddings from a frozen visual encoder and an aggregated tactile force vector $m_t=(X,Y,M)$ obtained by tracking the sensor's marker array; the refined window is then executed before the controller advances to the next segment of the action chunk.

What would settle it

Run the same dual-level pipeline on a strictly frozen generalist VLA with no task-specific fine-tuning on the cup, wipe, and peel tasks; if success rates fall to the base-VLA level, the reported gains come from the fine-tuning rather than from the dual-level tactile feedback.

Watch

Extended reading notes

Core claim

The central claim is that touch benefits manipulation through two complementary pathways, and that both are needed for reliable contact-rich behavior. A pretrained tactile-language model turns a short sequence of GelSight tactile images into linguistic descriptions of properties such as roughness and hardness; these descriptions are fed back to a vision-language task planner so it can choose objects or actions that are visually ambiguous. In parallel, an interpolant-based diffusion controller takes the action chunk produced by the base VLA as a source distribution and refines it toward expert actions, conditioned on visual state, robot proprioception, and a compact aggregated force vector estimated from marker displacements. The paper argues that this dual-level design lets a VLA gain tactile competence without retraining it on tactile data, and that removing either level substantially degrades performance.

Load-bearing premise

The load-bearing premise is that a VLA policy that has never been trained on touch still produces action chunks close enough to expert behavior for a small tactile-conditioned controller to refine, which the experiments approximate with a task-fine-tuned VLA rather than a strictly frozen one.

Editorial extensions

If this is right

  • A large VLA can gain tactile capability by adding a small separately trained controller, avoiding an expensive joint retraining pass over tactile data.
  • Semantic, language-form touch feedback lets an off-the-shelf vision-language planner reason about object properties that are invisible in RGB images.
  • Combining planning-level and control-level touch is predicted to outperform either level alone on contact-rich manipulation tasks.
  • Multi-modal conditioning on both vision and touch for the refinement controller is necessary for the best pick, placement, wiping, and peeling results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'without fine-tuning' claim is tested on a VLA that was first aligned on Franka data and then fine-tuned per task, so the reported gains may not transfer to a strictly frozen generalist VLA; that is my inference from the appendix details, not a paper claim.
  • The continuous numeric hardness and roughness scores from the tactile classifier may carry more usable information than discrete language labels when object differences are subtle, which the paper's appendix hints at but does not fully exploit.
  • Because the interpolant controller runs at 8 Hz while the tactile sensor streams at 25 Hz or higher, an event-triggered refinement scheme that reacts to contact transitions might extract additional benefit from the same touch signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes VLA-Touch, a modular system that adds tactile feedback to a frozen Vision-Language-Action (VLA) policy at two levels: a tactile-language model (Octopi) converts tactile images into linguistic descriptions that guide a GPT-4o task planner, and a diffusion-based interpolant controller (BRIDGeR) refines VLA-generated action chunks using visual embeddings and low-dimensional tactile force signals. The authors claim this dual-level design improves both task planning and contact-rich manipulation without fine-tuning the base VLA. Experiments on three real-world tasks (cup placement, sponge wiping, mango peeling) report higher success rates for the full system than for variants using only one tactile level, with ablations of visual and tactile inputs to the controller. The paper also documents its limitations, including sensor mismatch with the Octopi pretraining distribution and the controller's low control frequency.

Significance. If the central claim held, the modular 'frozen-VLA plus external touch' recipe would be a practical and general contribution to contact-rich manipulation, since it would avoid re-training large VLA backbones on tactile data. The paper is commendable for real-robot evaluation, open-sourced code, detailed appendices on prompts and training, and honest limitation statements. However, the significance is substantially tempered by an internal contradiction: the experiments fine-tune the base RDT model extensively, so the headline 'without fine-tuning' claim is untested. The planning comparison is also confounded by hand-specified force reference values, and all success-rate claims lack statistical analysis. These issues make the empirical contribution weaker than advertised.

major comments (3)
  1. [Abstract, Section 3, Appendix C, Algorithm 1] The abstract and Section 3 state that VLA-Touch enhances VLA policies 'without fine-tuning the base VLA,' but Appendix C (Data Collection) reports that RDT was first fine-tuned on unrelated Franka data for 100k steps (≈80 hours) and then fine-tuned on the collected task dataset for 20k steps per task (≈16 hours) before being used as the source policy. Algorithm 1 and Eq. (3) treat this fine-tuned checkpoint as π, so the evaluation does not test a truly frozen generalist VLA. This is an internal inconsistency in the paper's own experimental description; the reported gains in Tables 1 and 2 may depend on the task-specific fine-tuning rather than on the dual-level tactile feedback itself.
  2. [Section 4.2, Appendix A] The planning comparison between 'GPT-4o + Tactile Image' and 'GPT-4o + Octopi (Ours)' is confounded for the Cup task. In the Ours condition, the prompt includes hand-specified reference magnitudes ('the max force magnitude is around 0.55 for an empty cup and around 1.1 for a full cup') together with the numeric shear-force vector, while the raw-image condition receives only images. Since force estimation uses marker tracking rather than Octopi (as stated in Section 4.2), the improved force-inference success (90% vs 50%) may be attributable to the calibration references rather than to the linguistic or structured representation. Moreover, the force feedback is a numeric tuple, not a natural-language description, so the claim that 'linguistic descriptions are more usable by GPT-4o' is not cleanly supported by this experiment.
  3. [Tables 1 and 2, Section 4.3] All empirical comparisons are based on 20 trials per condition with no confidence intervals, error bars, or significance tests. The claimed improvements (e.g., '42%, 140%, and 67% higher task success rates' in Section 4.3) are derived from small counts: Cup Pick is 9/20 vs 12/20 (a difference of three successes) and Wipe full is 5/20 vs 12/20, where the binomial 95% confidence intervals overlap. Given that the paper's primary evidence is real-world success rates, the absence of statistical analysis leaves the central empirical claim unsubstantiated.
minor comments (6)
  1. [Abstract and Section 4.2] The abstract claims planning efficiency improves 'by up to 40%,' but no planning-efficiency metric is defined in Section 4.2, and the reported results are property-inference success rates (90% force, 75% hardness, etc.). Please clarify what 'planning efficiency' refers to and where the 40% figure comes from.
  2. [Section 4.2] The no-tactile planning baseline (GPT-4o with scene image only) is described as 'comparable to random guessing,' but the actual success counts are not provided; please report the quantitative results for this condition in Figure 3C or in the text.
  3. [Section 4.1 and Table 1] The base VLA is referred to both as 'RDT' and 'RDT-1B' inconsistently; please unify the nomenclature. Also clarify in the Table 1 caption that the 'w/o Touch' and 'w/o Vision' rows ablate inputs to the Interpolant controller only.
  4. [Appendix C, Eq. (3)] The training objective is written as an MSE loss against expert actions, but the BRIDGeR interpolant framework typically optimizes a velocity-matching or denoising objective; please specify the exact loss used in practice and how it relates to the interpolant SDEs in Eqs. (1)-(2).
  5. [Figure 4] The figure caption says '20 trails of peeling results' — this should be '20 trials.'
  6. [References] Reference [41] (TacNiq) lists an access date of September 8, 2025, which is later than the manuscript's July 2025 date; please correct the access date or the citation.

Circularity Check

1 steps flagged · score 2.0 of 10

No structural circularity; the empirical claims stand on real-world task success, but the cup force-planning evaluation supplies a threshold that is partly fitted to the task, and the two core modules are self-citations from the same lab.

  1. fitted input called prediction [Section 4.2 / Appendix A, Cup task 'Ours' prompt]
    "For reference, the max force magnitude is around 0.55 for an empty cup and around 1.1 for a full cup. | We use the marker tracking algorithm to get estimated force vectors and provide to GPT-4o along with baseline reference vectors. The reference values serve as calibration points when force estimates are relative or unnormalized, though they are less critical when methods report forces in absolute units (e.g., Newtons)."

    The cup planning evaluation measures whether GPT-4o correctly decides 'empty' versus 'full'. The method gives GPT-4o the exact separation threshold (0.55 for empty, 1.1 for full) inside the prompt, so the reported 90% 'force prediction' success is largely a threshold comparison on sensor measurements with a calibration key supplied in advance, rather than an independent inference produced by tactile feedback. The reference values are tied to the same task setup, making this one planning sub-task partly a fitted-input-as-prediction result. The other planning sub-tasks (roughness, hardness) and all manipulation results are externally measured and do not reduce to these reference values.

full rationale

The central derivation chain is otherwise self-contained: the interpolant controller is trained with supervised MSE against expert actions (Appendix C, Eq. 3), and its evaluation is physical task completion in Tables 1 and 2, so those results do not reduce to the controller's own training inputs. Octopi and BRIDGeR are prior work from the same lab, but this is not load-bearing circularity: Octopi is a pretrained module evaluated through external object-selection outcomes, and BRIDGeR provides a training framework rather than evidence for the paper's empirical claims. The Appendix C description of fine-tuning RDT for 100k Franka-alignment steps plus 20k per-task steps makes the abstract's 'without fine-tuning the base VLA' claim untested; that is an internal-validity inconsistency, not a circular reduction. The modest fitted-element in the cup force reference affects one of three planning sub-tasks and does not propagate to the manipulation or ablation conclusions. Overall, the paper has real independent empirical content, with only a minor calibration issue and self-citation presence, so a score of 2 is appropriate.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central results rely on pretrained Octopi and BRIDGeR modules from the same lab, a learned interpolant controller with unreleased weights, hand-set force calibration references, and untested assumptions about reliability of tactile language descriptions under a different gripper configuration. No new physical entities are introduced.

free parameters (2)
  • Interpolant controller weights theta = trained on DVLA, weights not released
    The controller's parameters are optimized with MSE loss (Eq. 3) on collected expert demonstrations, so the central control result depends on these fitted weights.
  • Force calibration references in planning prompt = empty cup approximately 0.55, full cup approximately 1.1
    Appendix A supplies these magnitudes to GPT-4o as references for interpreting force estimates; they are calibrated values derived from the task setup, not from an independent first-principles model.
assumptions (5)
  • domain assumption Octopi's tactile-language outputs are valid indicators of object hardness and roughness for the new gripper configuration
    High-level planning relies on these property scores to select objects (Section 4.2). The limitation section acknowledges the gripper setup differs from Octopi's pretraining, which compromises hardness interpretation.
  • domain assumption The aggregated force signal m_t=(X,Y,M) from GelSight marker tracking is a sufficient low-dimensional representation of contact for refinement
    The interpolant controller conditions only on this aggregated force, omitting per-marker contact geometry and high-frequency signals; Section 3.2 and Limitations.
  • domain assumption GPT-4o can reliably translate scene images plus linguistic tactile feedback into executable primitive actions
    Task planning success depends on the VLM following the prompt protocol (Appendix A); no detailed analysis of planner failures beyond task outcomes is provided.
  • standard math BRIDGeR stochastic interpolants can map VLA-generated action chunks to expert action distributions when conditioned on tactile and visual observations
    The method inherits the theoretical guarantees of interpolant diffusion from [44], which the paper does not re-derive; this is background methodology.
  • domain assumption The RDT-1B base policy, after task fine-tuning, provides a useful source action distribution for refinement
    Appendix C shows RDT was fine-tuned on task data, so the 'tactile refinement without fine-tuning' scenario is not actually tested; the source policy is not frozen.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback." pith.science (2026). https://pith.science/paper/UEJ26R4I

@misc{pith2026250717294,
  author       = {Pith},
  title        = {Pith review of: VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UEJ26R4I}},
  note         = {Machine review of arXiv:2507.17294}
}
read the original abstract

Tactile feedback is generally recognized to be crucial for effective interaction with the physical world. However, state-of-the-art Vision-Language-Action (VLA) models lack the ability to interpret and use tactile signals, limiting their effectiveness in contact-rich tasks. Incorporating tactile feedback into these systems is challenging due to the absence of large multi-modal datasets. We present VLA-Touch, an approach that enhances generalist robot policies with tactile sensing \emph{without fine-tuning} the base VLA. Our method introduces two key innovations: (1) a pipeline that leverages a pretrained tactile-language model that provides semantic tactile feedback for high-level task planning, and (2) a diffusion-based controller that refines VLA-generated actions with tactile signals for contact-rich manipulation. Through real-world experiments, we demonstrate that our dual-level integration of tactile feedback improves task planning efficiency while enhancing execution precision. Code is open-sourced at \href{https://github.com/jxbi1010/VLA-Touch}{this URL}.

Figures

Figures reproduced from arXiv: 2507.17294 by the authors.

Figure 1
Figure 1. VLA-Touch incorporates Dual-level Tactile Feedback for Planning and Manipulation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Experiments pipeline with images captured by the scene camera. Each task contains [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. A: Raw tactile images from three tasks requiring feedback on force, surface roughness, and hardness. B: Corresponding linguistic tactile descriptions generated by the Tactile-Language Model (Octopi) and GPT-4o’s responses. C: Task planning outcomes under three conditions: GPT-4o with no tactile input, with raw tactile images, and with linguistic tactile descriptions. Each condition was evaluated over 20 trials using… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: 20 trails of peeling results, demonstrating the qualitative effect of tactile feedback on peeling. Peel. The mango peeling task requires contact positioning and sustained pressure throughout the motion. RDT frequently slides without cutting (5/13 successful grasps) due…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Feeling the Unexpected: ResTacVLA for Contact-Rich Manipulation via Residual Tactile Representation

    cs.RO 2026-07 conditional novelty 6.5 of 10

    Residual tactile representations plus surprise-aware gating let VLA policies master contact-rich robot tasks that vision-only models fail.

  2. Demystifying When and Why VLAs Fail in Contact-Rich Tasks and How to Fix Them

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A noise-schedule change and time-aware force injection lift a VLA policy's average success on five contact-rich manipulation tasks from 41% to 66%.

  3. $N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A scaled tactile-native world-action model jointly predicts vision, touch, and action and outperforms vision-only baselines on contact-rich sim and real robot tasks.

  4. Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Future tactile prediction applied to intermediate action-expert features, rather than visual-language or final-action features, improves contact-rich manipulation in SmolVLA and π0.

  5. SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Success-only metrics overstate deformable-manipulation performance; tactile sensing raises Safety Success (e.g. 21.4%→35.6% on Object-Soft) while Goal Success stays comparable.

  6. Tactile Modality Fusion for Vision-Language-Action Models

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A FiLM-based tactile fusion method that conditions VLA visual features on frozen pretrained touch embeddings improves real-robot insertion success, speed, and force control relative to vision-only and concatenation baselines.

  7. TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Mechanics-aware future tactile prediction plus history and train-time isolation of future tokens raises real-robot contact-rich success from ~37.5% to 75% average.

  8. HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing

    cs.RO 2026-03 conditional novelty 5.0 of 10

    A vision-language-action robot policy that claims tactile-aware manipulation without tactile sensors at inference via reward-weighted flow matching and action distillation.

Reference graph

Works this paper leans on

47 extracted references · 29 canonical work pages · cited by 8 Pith papers

  1. [1]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine, Y . Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Retting...

    arXiv 2022

  2. [2]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=ZMnD6QZAE6

    work page 2024

  3. [3]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V . Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y . Lu, S. Levine, L. Lee, T.-W. E. Lee, I. Leal, Y . Kuang, D. Kalashnikov, R. Julia...

    work page 2023

  4. [4]

    O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid...

    arXiv

  5. [5]

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. RDT-1b: a diffusion foundation model for bimanual manipulation. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum? id=yAzN4tz7oI

    work page 2025

  6. [6]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. V...

    arXiv 2025

  7. [7]

    N. Funk, C. Chen, T. Schneider, G. Chalvatzaki, R. Calandra, and J. Peters. On the importance of tactile sensing for imitation learning: A case study on robotic match lighting, 2025. URL https://arxiv.org/abs/2504.13618

    arXiv 2025

  8. [8]

    Z. He, H. Fang, J. Chen, H.-S. Fang, and C. Lu. Foar: Force-aware reactive policy for contact-rich robotic manipulation. IEEE Robotics and Automation Letters, 10(6):5625–5632,

    work page

Show all 47 references
  1. [10]

    B. Ai, S. Tian, H. Shi, Y . Wang, C. Tan, Y . Li, and J. Wu. Robopack: Learning tactile-informed dynamics models for dense packing, 2024. URL https://arxiv.org/abs/2407.01418

    2024 arXiv

  2. [11]

    E. K. Miller and J. D. Cohen. An integrative theory of prefrontal cortex function. Annual review of neuroscience, 24(1):167–202, 2001

    2001

  3. [12]

    L. Tame, F. Pavani, C. Papadelis, A. Farne, and C. Braun. Early integration of bilateral touch in the primary somatosensory cortex. Human brain mapping, 36(4):1506–1523, 2015

    2015

  4. [13]

    J. C. Culham and K. F. Valyear. Human parietal cortex in action. Current opinion in neurobiology, 16(2):205–212, 2006

    2006

  5. [14]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Manj...

    2023

  6. [15]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics:...

    2024

  7. [16]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. π0: A vision-l...

    2024 arXiv

  8. [17]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-e: An emb...

    2023

  9. [18]

    X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y . Jing, W. Zhang, H. Liu, H. Li, and T. Kong. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum...

    2024

  10. [19]

    Cheang, G

    C.-L. Cheang, G. Chen, Y . Jing, T. Kong, H. Li, Y . Li, Y . Liu, H. Wu, J. Xu, Y . Yang, H. Zhang, and M. Zhu. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation, 2024. URL https://arxiv.org/abs/2410.06158

    2024 arXiv

  11. [20]

    Bjorck, F

    NVIDIA, :, J. Bjorck, F. Casta ˜neda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y . L. Tan...

    2025 arXiv

  12. [21]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An emb...

    2023 arXiv

  13. [22]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, October 2023

    2023

  14. [23]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut,...

    2024

  15. [24]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Zhao, Y . Huang, A. Dai, H. Y...

    2024

  16. [25]

    H.-S. Fang, H. Fang, Z. Tang, J. Liu, J. Wang, H. Zhu, and C. Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023. 12

    2023

  17. [26]

    Jones, O

    J. Jones, O. Mees, C. Sferrazza, K. Stachowicz, P. Abbeel, and S. Levine. Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding, 2025. URL https://arxiv.org/abs/2501.04693

    2025 arXiv

  18. [27]

    P. Hao, C. Zhang, D. Li, X. Cao, X. Hao, S. Cui, and S. Wang. Tla: Tactile-language-action model for contact-rich manipulation. arXiv preprint arXiv:2503.08548, 2025

    2025 arXiv

  19. [28]

    J. Zhao, Y . Ma, L. Wang, and E. Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=KXsropnmNI

    2024

  20. [29]

    Higuera, A

    C. Higuera, A. Sharma, C. K. Bodduluri, T. Fan, P. Lancaster, M. Kalakrishnan, M. Kaess, B. Boots, M. Lambeta, T. Wu, and M. Mukadam. Sparsh: Self-supervised touch representations for vision-based tactile sensing. In 8th Annual Conference on Robot Learning,

  21. [30]

    R. Feng, J. Hu, W. Xia, T. Gao, A. Shen, Y . Sun, B. Fang, and D. Hu. Learning unified static-dynamic representation across multiple visuo-tactile sensors. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=XToAemis1h

    2025

  22. [31]

    GelSight Mini Tactile Sensor

    GelSight Inc. GelSight Mini Tactile Sensor. https://www.gelsight.com/ gelsightmini/, 2023. Accessed: 2025-04-29

    2023

  23. [32]

    Lambeta, P.-W

    M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V . R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, D. Jayaraman, and R. Calandra. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics a...

    2020 doi

  24. [33]

    Zhang, Y

    S. Zhang, Y . Yang, F. Sun, L. Bao, J. Shan, Y . Gao, and B. Fang. A compact visuo-tactile robotic skin for micron-level tactile perception. IEEE Sensors Journal, 24(9):15273–15282,

  25. [34]

    W. K. Do, B. Jurewicz, and M. Kennedy. Densetact 2.0: Optical tactile sensor for shape and force reconstruction. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12549–12555, 2023. doi:10.1109/ICRA48891.2023.10161150

    2023

  26. [35]

    S. Yu, K. Lin, A. Xiao, J. Duan, and H. Soh. Octopi: Object property reasoning with large tactile-language models. In Proceedings of Robotics: Science and Systems, 2024

    2024

  27. [36]

    doi:10.1109/JSEN.2024.3376574

    2024

  28. [37]

    Huang, Y

    B. Huang, Y . Wang, X. Yang, Y . Luo, and Y . Li. 3d vitac:learning fine-grained manipulation with visuo-tactile sensing. In Proceedings of Robotics: Conference on Robot Learning(CoRL), 2024

    2024

  29. [38]

    H. Xue, J. Ren, W. Chen, G. Zhang, Y . Fang, G. Gu, H. Xu, and C. Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation, 2025. URL https://arxiv.org/abs/2503.02881

    2025 arXiv

  30. [39]

    L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg. A touch, vision, and language dataset for multimodal alignment. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview.net...

    2024

  31. [40]

    uskin patch - tactile sensor, 2024

    XELA Robotics. uskin patch - tactile sensor, 2024. URL https://www.xelarobotics. com/uskin-patch. Accessed: 2025

    2024

  32. [41]

    TAC-02 Robotic Finger Development Kit, 2024

    TacNiq. TAC-02 Robotic Finger Development Kit, 2024. URL https://www.tacniq.ai/ tac-02-robotic-finger-dev-kit . Accessed: September 8, 2025

    2024

  33. [42]

    J. Zhao, N. Kuppuswamy, S. Feng, B. Burchfiel, and E. Adelson. Polytouch: A robust multi-modal tactile sensor for contact-rich manipulation using tactile-diffusion policies, 2025. URL https://arxiv.org/abs/2504.19341. 13

    2025 arXiv

  34. [43]

    S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  35. [44]

    K. Chen, E. Lim, K. Lin, Y . Chen, and H. Soh. Don’t start from scratch: Behavioral refinement via interpolant-based policy diffusion. arXiv preprint arXiv:2402.16075, 2024. 14 Appendix A Prompt Examples For all tasks, we first give the VLM the following global context: System...

    2024 arXiv

  36. [45]

    Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, and H. J. Ritter. A review of tactile information: Perception and action through touch. IEEE Transactions on Robotics , 36(6): 1619–1634, 2020

    2020

  37. [48]

    Ours User: There is a cup in the image

    **Action:** Place the cup back in its original position. Ours User: There is a cup in the image. Step by step, identify whether the cup is full or empty. If it is not empty, put it onto the plate. Robot: 1. Action: Grip the cup gently to lift it slightly. - Information Needed:...

  38. [2024]

    URL https://openreview.net/forum?id=xYJn2e1uu8

  39. [2025]

    doi:10.1109/LRA.2025.3560871

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.