Pith. sign in

REVIEW 9 cited by

Sparsh: Self-supervised touch representations for vision-based tactile sensing

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.24090 v1 pith:H7T5QBRE submitted 2024-10-31 cs.RO

Carolina Higuera , Akash Sharma , Chaithanya Krishna Bodduluri , Taosha Fan , Patrick Lancaster , Mrinal Kalakrishnan , Michael Kaess , Byron Boots
show 3 more authors
Mike Lambeta Tingfan Wu Mustafa Mukadam
This is my paper · ORCID
classification cs.RO
keywords tactilesensorssparshmodelstasktouchvision-basedimages
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advances in robot manipulation as they markedly complement vision, yet solutions today often rely on task and sensor specific handcrafted perception models. Collecting real data at scale with task centric ground truth labels, like contact forces and slip, is a challenge further compounded by sensors of various form factor differing in aspects like lighting and gel markings. To tackle this we turn to self-supervised learning (SSL) that has demonstrated remarkable performance in computer vision. We present Sparsh, a family of SSL models that can support various vision-based tactile sensors, alleviating the need for custom labels through pre-training on 460k+ tactile images with masking and self-distillation in pixel and latent spaces. We also build TacBench, to facilitate standardized benchmarking across sensors and models, comprising of six tasks ranging from comprehending tactile properties to enabling physical perception and manipulation planning. In evaluations, we find that SSL pre-training for touch representation outperforms task and sensor-specific end-to-end training by 95.1% on average over TacBench, and Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, indicating the merits of learning in latent space for tactile images. Project page: https://sparsh-ssl.github.io/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Prediction targets, not inputs, decide which physical parameters a latent world model acquires; a certified-recoverable drag parameter stays unlearned under every deterministic prediction objective tested.

  2. Feeling the Unexpected: ResTacVLA for Contact-Rich Manipulation via Residual Tactile Representation

    cs.RO 2026-07 conditional novelty 6.5 of 10

    Residual tactile representations plus surprise-aware gating let VLA policies master contact-rich robot tasks that vision-only models fail.

  3. Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A self-supervised multimodal encoder trained with vision, proprioception, and force yields a vision-only latent that recovers end-effector state and force above vision baselines on RH20T, with modest absolute force accuracy.

  4. OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A policy-agnostic two-stage real-world RL method learns tactile residual corrections on frozen visual policies, lifting contact-rich task success from 5–40% to 85–100% in under 80 minutes.

  5. TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    TouchThinker introduces a 1M-scale multi-source tactile dataset and action-aware modeling to scale commonsense reasoning from tactile observations, reporting competitive performance on new and existing benchmarks.

  6. Tactile Modality Fusion for Vision-Language-Action Models

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A FiLM-based tactile fusion method that conditions VLA visual features on frozen pretrained touch embeddings improves real-robot insertion success, speed, and force control relative to vision-only and concatenation baselines.

  7. Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

    cs.RO 2025-12 unverdicted novelty 6.0 of 10

    DreamTacVLA grounds VLA models in contact physics by aligning multi-scale vision-tactile inputs and predicting future tactile states, reaching up to 95% success on contact-rich tasks.

  8. 3D Cal: An Open-Source Software Library for Depth Reconstruction on Vision-Based Tactile Sensors

    cs.RO 2025-11 conditional novelty 6.0 of 10

    3D Cal repurposes a 3D printer as an automated calibration rig and trains a lightweight CNN, TouchNet, to reconstruct depth maps for DIGIT and GelSight Mini.

  9. TensorTouch: Calibration of Tactile Sensors for High Resolution Stress Tensor and Deformation for Dexterous Manipulation

    cs.RO 2025-06 conditional novelty 6.0 of 10

    TensorTouch converts optical tactile sensor images into dense stress tensor, deformation, and contact force fields using finite-element simulation and a hierarchical vision transformer, and uses these fields for selec...

Pith tools