Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read REASSEMBLE offers 4,551 demonstrations of contact-rich assembly and disassembly on the NIST Task Board #1, with synchronized event camera, force-torque, audio, and multi-view RGB data, annotated for temporal action segmentation, motion…

desk verdict A genuinely useful contact-rich manipulation dataset with event cameras, but the headline 4,551 demonstration count counts action segments, not task executions, and the benchmarks are too thin to stand as the paper's evaluation. read the letter →

arxiv 2502.05086 v2 pith:NLXY4NAI submitted 2025-02-07 cs.RO

Daniel Sliwowski , Shail Jadav , Sergej Stanovcic , Jedrzej Orbik , Johannes Heidersberger , Dongheui Lee This is my paper · ORCID
classification cs.RO
keywords multimodalmanipulationdatasetcontact-richassemblydisassemblyeventcameraforce-torquesensingtemporalactionsegmentationmotionpolicylearninganomalydetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents REASSEMBLE, a public dataset of 4,551 demonstrations (4,035 successful) of pick, insert, remove, and place actions on 17 objects from the NIST Assembly Task Board #1. The central claim is that the dataset captures the physical reality of contact-rich assembly and disassembly through synchronized high-frequency sensors that most manipulation datasets lack: an event camera, a six-axis force-torque sensor, three microphones, multi-view RGB cameras, and robot proprioception. The authors argue that existing datasets concentrate on simple rearrangement or short-horizon skills, leaving long-horizon tasks with tight tolerances and force-relevant contact dynamics under-supplied. If REASSEMBLE delivers on that claim, it gives the field a common test bed for learning policies that need to feel an insertion succeed, detect a jam, and segment long task sequences into reusable skills. The value of the dataset depends on whether the recorded contacts, forces, and failures are faithful enough to transfer to real assembly lines and to the official NIST board.

What carries the argument

The carrying mechanism is the force-feedback teleoperation rig: a human operator drives a 7-DoF Cartesian impedance controller through a haptic master while the wrist force-torque signal is scaled back to the operator's hand. The control law $\tau_{\mathrm{imp}} = J^\top(K_x e_\xi + D \dot{e}_\xi)$ and the nullspace torque that keep the arm away from singularities appear in the paper as the precise path from demonstrated contact force to recorded data. The standardized NIST Assembly Task Board #1 supplies the contact-rich geometry—pegs, gears, BNC, USB, Ethernet, waterproof and D-SUB connectors—and the annotation pipeline (narrated speech transcribed by Whisper, corrected and manually validated, then hierarchically labelled into four actions and nine skills) turns the raw streams into the three task labels.

What would settle it

Run the released gear-insertion policy on an unmodified, original-tolerance acrylic NIST Task Board #1 with the official Ethernet and USB holders and the nuts included; if insertion success falls below the reported 7 out of 10, or if the measured insertion force profiles differ systematically from the dataset's mean wrench profiles such as the contact and alignment phase between 20% and 75% progress, then the board modifications have altered the contact physics the dataset claims to capture.

Watch

Extended reading notes

Core claim

The paper's central claim is that REASSEMBLE answers the shortage of long-horizon, contact-rich manipulation data by pairing a standardized benchmark with sensors that record what contact actually feels, sounds, and looks like at high temporal resolution. The dataset has 4,551 action demonstrations (4,035 successful) across four actions—pick, insert, remove, place—over 17 objects on the NIST Assembly Task Board #1, producing 68 balanced action-object pairs and 10,195 low-level skill labels. Its distinct contribution is the sensor set: multi-view RGB, robot proprioception, three microphones, a six-axis force-torque sensor, and an externally mounted event camera, which the authors state is the first event-camera stream in a manipulation-focused dataset. On top of the raw data, the paper supplies hierarchical annotations (high-level actions and low-level skills), success flags, and failure demonstrations, and it reports three benchmarks to show the data are usable: DiffAct temporal action segmentation with 61.5% accuracy and 44.1% F1@50, a DMP-based motion policy that reproduces gear assembly and disassembly at 8 out of 10, 7 out of 10, 8 out of 10, and 10 out of 10 success for pick, insert, remove, and place, and a ConditionNET-style execution monitor with 96% accuracy, precision, and recall.

Load-bearing premise

The load-bearing premise is that the demonstrations teach real contact-rich physics even though the benchmark board was modified—acrylic base swapped for matte PVC, Ethernet and USB holders redesigned, and objects held upright in custom 3D-printed fixtures—so the tolerances and contact forces may not match the official NIST board or industrial parts.

Editorial extensions

If this is right

  • REASSEMBLE gives temporal action segmentation researchers a benchmark with 69 action classes and synchronized force-torque, audio, and event streams, where the reported visual-only DiffAct baseline (F1@50 of 44.1%) has clear room for multimodal fusion to improve.
  • Motion policies can be evaluated against the dataset's contact signatures, such as the reported mean wrench profiles where the vertical force drops as a waterproof plug aligns and snaps into place.
  • With 516 failed demonstrations and per-action success flags, anomaly detection models can be trained to recognize unsatisfied preconditions before they cause damage, using the 96%-accurate ConditionNET-style monitoring setup as a starting point.
  • Because assembly and disassembly of the same objects are both recorded on the same board, the dataset supports task inversion learning: a model that learns to insert a gear can be evaluated on removing it using the same force thresholds and spiral search strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own benchmarks do not ablate the event camera, so the strongest test of its novelty claim is to train the same temporal action segmentation or anomaly detection model with and without event data, holding all other inputs fixed.
  • The board modifications are the main transfer risk: if the matte PVC base and vertical fixtures change friction or compliance, policies learned on REASSEMBLE may need domain adaptation before they work on the unmodified official board or on factory equipment.
  • The low visual-only F1@50 suggests the 69 classes overwhelm purely video-based segmentation; a natural next experiment is to feed force-torque and audio into the same DiffAct model and measure the accuracy gain.
  • The four-verb action set was chosen to keep human labelling manageable, so extending the same protocol to the other NIST task boards, which the authors list as future work, would reveal whether the annotation scheme scales to cable routing and flexible belts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces REASSEMBLE, a multimodal dataset for contact-rich robotic assembly and disassembly built around a modified NIST Assembly Task Board #1. Data were collected via haptic teleoperation of a Franka FR3 and include RGB video, event camera streams, force–torque measurements, audio, and robot proprioception. The dataset provides hierarchical annotations (four high-level actions and nine low-level skills), success/failure labels, and supports temporal action segmentation, motion policy learning, anomaly detection, and task inversion learning. The authors describe the sensor setup, calibration, teleoperation equations, HDF5 storage format, a visualization tool, and preliminary benchmark results with DiffAct, DMPs, and ConditionNET. The paper claims 4,551 demonstrations, 4,035 successful, spanning 781 minutes, and is positioned as a new public resource for long-horizon contact-rich manipulation.

Significance. If the dataset is properly scoped and the claims are corrected, REASSEMBLE would be a useful community resource: it combines modalities that are rare together in manipulation datasets (event camera, wrist force–torque, and audio) with hierarchical annotations and explicit failure labels. The authors provide concrete infrastructure: public HDF5 files, a documented data structure, calibration procedures, an annotation validation tool, and code for loading data. These are concrete strengths that go beyond many dataset papers. The preliminary benchmarks show that the data can support the three advertised tasks. However, the headline demonstration count is inflated, the board modifications weaken the link to the official NIST benchmark, and the benchmark protocols are too thin to support the broader generalization claims.

major comments (3)
  1. [Abstract, Table I, Section IV-A, Fig. 1] The central claim of '4,551 demonstrations' is contradicted by the paper's own statistics, which show that this number counts action segments, not full task demonstrations. Section IV-A states 'The REASSEMBLE dataset contains a total of 4,551 actions, decomposed into 10,195 skills,' and Figure 1 labels the same number as '4551 high level demonstrations.' If 4,551 were complete assembly/disassembly recordings, the 781 minutes would imply an average of about 10.3 seconds per demonstration, which is incompatible with the long-horizon tasks and with the reported median of 36 action segments per video in Section V-A. The dataset therefore contains on the order of a hundred or a few hundred full recordings, each containing many action segments. As written, the abstract and Table I compare an action-segment count against other datasets' full-demonstration counts, overstating the amount of independent task experience a user obtains. This issue is load-bearing because the 'comprehensive dataset' claim rests on this number. Please report the number of full task recordings separately from the number of action demonstrations, and revise the abstract, Table I, and Figure 1 accordingly.
  2. [Section III-A] The paper describes substantial modifications to the NIST Task Board #1: the acrylic base was replaced with matte white PVC, the Ethernet and USB holders were redesigned, and 3D-printed vertical holders were added for pegs and connectors. These changes may alter clearances, contact stiffness, friction, and required insertion forces relative to the official benchmark, which matters because the paper repeatedly grounds the dataset's value in being 'built around the NIST Assembly Task Board 1 benchmark' and in facilitating deployment across research institutes. The modifications are reasonable for single-arm teleoperation, but the paper should either quantify how the key physical parameters (e.g., peg-hole clearances, connector insertion forces, board compliance) compare with the official board, or explicitly scope the dataset as 'NIST-inspired' rather than claiming benchmark equivalence. Without this, the generalization and transferability implied by the NIST framing is not supported.
  3. [Section V] The benchmark evidence is too thin to support the claim that the dataset reliably supports all three advertised tasks. The DiffAct TAS experiment reports no train/validation split, no number of training videos, and no variance over seeds or runs; the ConditionNET experiment reports 96% accuracy, precision, and recall but no split details, no dataset size, and no baseline; the DMP policy experiment covers only the large gear with 10 trials per action. For a dataset paper, the benchmarks need not be state-of-the-art, but they must be reproducible and must give the reader a realistic view of variability. Please provide the exact data splits, the subset of recordings used, the number of random seeds or repetitions, and the standard deviation of each metric, or explicitly label these results as illustrative sanity checks rather than benchmark evaluations.
minor comments (6)
  1. [Appendix B, Fig. 13] The HDF5 structure lists 'measured_torque' with shape N_mt x 7 but describes it as (x, y, z), and 'velocity' with shape N_vels x 7 but six listed components (x, y, z, omega, gamma, theta). Please correct the dimensions and component lists to match the actual saved arrays and the six-axis torque channel count.
  2. [Table I] The row for REASSEMBLE lists '# Demos 4k' and compares it directly with full-demonstration counts of other datasets; even after the counting issue in the major comments is fixed, the table should state whether the number refers to action segments or full task recordings and should avoid mixing the two notions.
  3. [Section V-A] The DiffAct baseline uses the default hyperparameters tuned for 50Salads; this choice should be stated as a limitation or justified, because the difference in action count, segment length, and modality mix may make those defaults suboptimal for REASSEMBLE.
  4. [Abstract] The final sentence contains a grammatical error: 'The REASSEMBLE will be a valuable resource' should read 'REASSEMBLE will be a valuable resource.'
  5. [Section IV-D] The sentence 'We observed that some errors occur in the transcriptions; for example, "D-SUB" is often recognized as "the sub."' is missing a period at the end and appears mid-paragraph; please fix the punctuation.
  6. [Section III-A and Figure 15] The list of objects in the appendix says '1 M16 Nut' but Section III-A states that all three nuts (M12, M8, and M4) were excluded while the M16 nut is included; please clarify that the dataset contains the M16 nut only and update the object count explanation accordingly.

Circularity Check

0 steps flagged · score 1.0 of 10

No meaningful circularity: REASSEMBLE is an empirical dataset contribution; self-citations to ConditionNET and M2R2 are illustrative benchmarks, not load-bearing derivations.

full rationale

REASSEMBLE makes no derived prediction: its central claim is the existence, content, and annotation of a multimodal dataset, supported by an external standard (NIST Task Board #1) and recorded sensor streams. The benchmark sections train standard or prior models (DiffAct, DMPs, ConditionNET) on the released data and report empirical outcomes; none of these outcomes is obtained by construction from the dataset statistics. The two self-citations ([23] ConditionNET and [44] M2R2) are used as methodological baselines or preliminary demonstrations for downstream tasks; the dataset's validity does not depend on either work, so they are not load-bearing. The limitations section (Sec. VI) explicitly acknowledges object similarity and missing depth information, and the modified-board disclosure (Sec. III-A) is a transparency statement rather than a circular justification. The only notable internal-consistency concern is that the abstract and Table I call 4,551 items 'demonstrations' while Sec. IV-A defines them as 4,551 actions decomposed into 10,195 skills; this is a reporting and statistical-semantics issue, not a circular reduction, and it does not affect the dataset's independent evidentiary status.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is the dataset's existence and content; it rests on data collection assumptions rather than fitted parameters. Teleoperation and DMP benchmark parameters are listed for completeness.

free parameters (5)
  • Teleoperation scale S = diag(4.5,4.5,4.5)
    Eq. (1) maps haptic device motion to robot workspace; chosen by hand for the specific setup.
  • Force feedback scaling = 0.35
    Eq. (7) scales sensed forces for haptic feedback; chosen by hand.
  • Nullspace weighting coefficients = alpha, beta, gamma (values not reported)
    Eq. (9) combines manipulability, joint-limit, and velocity damping objectives; chosen by hand.
  • DMP basis function weights = Learned per skill via least squares
    Eq. (14) fits forcing term to demonstrations in the MPL benchmark; not central to dataset claim but fitted to data.
  • Force thresholds for spiral insertion = Not reported numerically
    Section V-B says thresholds are learned from the dataset; used in policy execution.
assumptions (4)
  • standard math Rigid-body robot kinematics and Jacobian relations in Eqs. (5) and (8) are correct for the Franka FR3.
    Used to compute impedance control torques and nullspace torques in teleoperation; standard results.
  • domain assumption Modifications to NIST Task Board #1 (PVC base, redesigned holders, 3D-printed fixtures) preserve the original benchmark's contact-rich task characteristics.
    Section III-A states minor modifications for compatibility; if tolerances or geometry changed significantly, demonstrations would not represent the standardized benchmark.
  • domain assumption Haptic teleoperation with force feedback yields demonstrations representative of natural human assembly and disassembly behavior, including failures.
    Section III-C relies on the operator feeling interaction forces; if the teleoperation distorts motion or force profiles, downstream policies learned from the dataset may not transfer.
  • domain assumption Whisper-based speech transcription plus manual validation produces correct action and skill annotations.
    Section III-D describes correcting transcription errors and validating labels; annotation correctness is load-bearing for TAS benchmarks and success labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly." pith.science (2026). https://pith.science/paper/NLXY4NAI

@misc{pith2026250205086,
  author       = {Pith},
  title        = {Pith review of: REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLXY4NAI}},
  note         = {Machine review of arXiv:2502.05086}
}
read the original abstract

Robotic manipulation remains a core challenge in robotics, particularly for contact-rich tasks such as industrial assembly and disassembly. Existing datasets have significantly advanced learning in manipulation but are primarily focused on simpler tasks like object rearrangement, falling short of capturing the complexity and physical dynamics involved in assembly and disassembly. To bridge this gap, we present REASSEMBLE (Robotic assEmbly disASSEMBLy datasEt), a new dataset designed specifically for contact-rich manipulation tasks. Built around the NIST Assembly Task Board 1 benchmark, REASSEMBLE includes four actions (pick, insert, remove, and place) involving 17 objects. The dataset contains 4,551 demonstrations, of which 4,035 were successful, spanning a total of 781 minutes. Our dataset features multi-modal sensor data, including event cameras, force-torque sensors, microphones, and multi-view RGB cameras. This diverse dataset supports research in areas such as learning contact-rich manipulation, task condition identification, action segmentation, and task inversion learning. The REASSEMBLE will be a valuable resource for advancing robotic manipulation in complex, real-world scenarios. The dataset is publicly available on our project website: https://tuwien-asl.github.io/REASSEMBLE_page/.

Figures

Figures reproduced from arXiv: 2502.05086 by the authors.

Figure 1
Figure 1. Overview of the REASSEMBLE dataset. In REASSEMBLE, we focus on creating a dataset for contact-rich manipulation tasks. We leverage the well-established NIST Assembly Task Board #1 [1] to facilitate deployment of learned algorithms across different research institutes. The dataset includes data from various modalities, such as RGB cameras and robot proprioception, which are common in other works. Additionally, we inc… view at source ↗
Figure 2
Figure 2. Overview of the sensor placement. We use two external and one wrist-mounted RGB cameras (marked in orange). Additionally, we use an externally mounted event camera (in blue), three micro￾phones (in yellow), and one wrist-mounted force/torque (F/T) sensor (in red). The omega.6 haptic teleoperation device is also visible. Event cameras are emerging as critical sensors in robotics, providing a fundamentally different d… view at source ↗
Figure 3
Figure 3. , where the robot nudges a stuck peg. Once dislodged, the peg’s movement is clearly captured in the event data. Event camera information has also been successfully used in tasks such as slip detection [33], and has shown improved performance in human action recognition compared to using only RGB images [34]. Motivated by these findings, we incorporated event camera data into REASSEMBLE’s multi￾modal framework. (a) B… view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Sankey diagram showing the hierarchical structure and how [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Number of demonstrations of each action-object pair. In REASSEMBLE, we have 4 actions: pick, insert, remove, and place, and 17 objects, resulting in 68 unique action-object pairs. The number of executions of each unique action is almost equal, making it a balanced data…
Figure 7
Figure 7. Figure 7: Number of successful and unsuccessful demonstrations for each action-object pair. In REASSEMBLE, we also annotate the success of each action demonstration. This can serve as a proxy for the action difficulty. The insert action is the most difficult, as it requires the …
Figure 8
Figure 8. Figure 8: Approximate interaction point for each of the actions. The robot is placed at the point (0,0), facing the positive x-direction. REASSEMBLE has a large variety in the interaction points for each action. the mean force and torque values along with their standard deviatio…
Figure 9
Figure 9. Figure 9: Mean and standard deviation of force and torque measurements for selected actions. To analyze the temporal evolution of each action, we normalize all demonstrations by their duration and compute the average wrench profile over the action progress. Distinct patterns eme…
Figure 10
Figure 10. Figure 10: Temporal action segmentation results. In red, we highlight instances where the "Pick" action was not predicted by DiffAct. In blue, we mark areas where similar objects were confused. In this case, DiffAct confused "round peg 1" with "square peg 1" and "square peg 2." …
Figure 11
Figure 11. Figure 11: Large Gear assembly & disassembly The figure illustrates the trajectories generated by the DMP framework for robotic assembly and disassembly of the large gear, including Pick, Insert, Remove, and Place motions. These trajectories are executed using an impedance contr…
Figure 12
Figure 12. Figure 12: Example snapshots from the execution monitoring exper￾iments. On the left, the observation just before starting the execution of the approach primitive. In the middle, the moment after the gear is grasped and before the lift primitive is executed. On the right, the si…
Figure 13
Figure 13. Figure 13: shows the structure of a single demonstration file in the REASSEMBLE dataset. Each demonstration is stored in an individual file, identified by the date and time the demonstration began. The HDF5 (.h5) format distinguishes between two types of structures: datasets, wh…
Figure 14
Figure 14. Figure 14: Snapshot from the dataset visualization tool. Best viewed in color. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Objects and connectors used in the REASSEMBLE dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    RoboProcessBench is a new benchmark decomposing process-aware understanding into static monitoring and dynamic reasoning across 12 question families, with evaluations showing VLM limitations but post-training gains on...

  2. AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly

    cs.RO 2026-04 unverdicted novelty 6.0 of 10

    AssemLM uses a specialized point cloud encoder inside a multimodal LLM to reach state-of-the-art 6D pose prediction for assembly tasks, backed by a new 900K-sample benchmark called AssemBench.

  3. RoboLight: A Dataset with Linearly Composable Illumination for Robotic Manipulation

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A dataset that records identical robot manipulation tasks under 14 controlled lighting conditions and uses HDR linearity to synthesize 196,000 additional lighting-varied episodes.

  4. DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

    cs.RO 2026-02 conditional novelty 6.0 of 10

    A decoupled multimodal diffusion transformer with a LoRA tactile adapter improves real-world bimanual manipulation success by 21 percentage points over a diffusion-policy baseline; a new 50-hour tactile bimanual datas...

  5. Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective

    cs.AI 2025-09 conditional novelty 4.0 of 10

    Agent spatial intelligence is organized into six neuroscience-inspired modules, and the field is reviewed through that lens without any experimental validation.

Reference graph

Works this paper leans on

50 extracted references · 41 canonical work pages · cited by 5 Pith papers

  1. [1]

    Performance measures to benchmark the grasping, manipulation, and assembly of deformable objects typical to manufacturing applications,

    K. Kimble, J. Albrecht, M. Zimmerman, and J. Falco, “Performance measures to benchmark the grasping, manipulation, and assembly of deformable objects typical to manufacturing applications,” Frontiers in Robotics and AI , vol. 9, p. 999348, 2022

    work page 2022

  2. [2]

    Literature review of deep learning research areas,

    M. Yapıcı, A. Tekerek, and N. Topaloglu, “Literature review of deep learning research areas,” Gazi Journal of Engineering Sciences , vol. 5, pp. 188–215, 12 2019

    work page 2019

  3. [3]

    Diffusion policy: Visuomotor policy learning via ac- tion diffusion,

    C. Chi et al., “Diffusion policy: Visuomotor policy learning via ac- tion diffusion,” The International Journal of Robotics Research , p. 02783649241273668, 2023

    work page 2023

  4. [4]

    RT-1: Robotics Transformer for Real-World Control at Scale,

    A. Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale,” in Proceedings of Robotics: Science and Systems , Daegu, Republic of Korea, 7 2023

    work page 2023

  5. [5]

    RT-2: Vision-language-action models transfer web knowledge to robotic control,

    B. Zitkovich et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” in 7th Annual Conference on Robot Learning, 2023

    work page 2023

  6. [6]

    Do as i can, not as i say: Grounding language in robotic affordances,

    B. Ichter et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in 6th Annual Conference on Robot Learning , 2022

    work page 2022

  7. [7]

    Sayplan: Grounding large language models using 3d scene graphs for scalable task planning,

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. D. Reid, and N. Suenderhauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable task planning,” CoRR, 2023

    2023

  8. [8]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” pp. 8469–8488, 2023

    work page 2023

Show all 50 references
  1. [9]

    Learning agile robotic locomotion skills by imitating animals,

    X. B. Peng, E. Coumans, T. Zhang, T.-W. E. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” in Robotics: Science and Systems , 07 2020

    2020

  2. [10]

    Real-Time Anomaly Detection and Reactive Planning with Large Language Models,

    R. Sinha, A. Elhafsi, C. Agia, M. Foutter, E. Schmerling, and M. Pavone, “Real-Time Anomaly Detection and Reactive Planning with Large Language Models,” in Proceedings of Robotics: Science and Systems , Delft, Netherlands, 7 2024

    2024

  3. [11]

    Tacdiffusion: Force-domain diffusion policy for precise tactile manipulation,

    Y . Wu, Z. Chen, F. Wu, L. Chen, L. Zhang, Z. Bing, A. Swikir, A. Knoll, and S. Haddadin, “Tacdiffusion: Force-domain diffusion policy for precise tactile manipulation,” arXiv preprint arXiv:2409.11047 , 2024

    2024 arXiv

  4. [12]

    Hierarchical hybrid learning for long-horizon contact-rich robotic assembly,

    J. Sun, A. Curtis, Y . You, Y . Xu, M. Koehle, L. Guibas, S. Chitta, M. Schwager, and H. Li, “Hierarchical hybrid learning for long-horizon contact-rich robotic assembly,” arXiv preprint arXiv:2409.16451 , 2024

    2024 arXiv

  5. [13]

    Accelerating reinforcement learning with learned skill priors,

    K. Pertsch, Y . Lee, and J. Lim, “Accelerating reinforcement learning with learned skill priors,” in Conference on robot learning . PMLR, 2021, pp. 188–204

    2021

  6. [14]

    BC-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “BC-z: Zero-shot task generalization with robotic imitation learning,” in 5th Annual Conference on Robot Learning , 2021

    2021

  7. [15]

    Bridgedata v2: A dataset for robot learning at scale,

    H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen- Estruch, A. W. He, V . Myers, M. J. Kim, M. Du et al., “Bridgedata v2: A dataset for robot learning at scale,” in Conference on Robot Learning. PMLR, 2023, pp. 1723–1736

    2023

  8. [16]

    Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation,

    M. Heo, Y . Lee, D. Lee, and J. J. Lim, “Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation,” The International Journal of Robotics Research , p. 02783649241304789, 2023

    2023

  9. [17]

    DROID: A Large-Scale In-The-Wild Robot Ma- nipulation Dataset,

    A. Khazatsky et al., “DROID: A Large-Scale In-The-Wild Robot Ma- nipulation Dataset,” July 2024

    2024

  10. [18]

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot,

    H Fang et al., “Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot,” inIEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 653–660

    2024

  11. [19]

    Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling,

    Y . Gao, S. S. Vedula, C. E. Reiley, N. Ahmidi, B. Varadarajan, H. C. Lin, and G. D. Hager, “Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling,” in MICCAI Workshop: M2CAI, vol. 3, no. 2014, 9 2014, p. 3

    2014

  12. [20]

    Combining embedded accelerometers with computer vision for recognizing food preparation activities,

    S. Stein and S. J. McKenna, “Combining embedded accelerometers with computer vision for recognizing food preparation activities,” in Pro- ceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing , ser. UbiComp ’13. New York, NY , USA: Assoc...

    2013

  13. [21]

    Assembly101: A large-scale multi-view video dataset for understanding procedural activities,

    F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao, “Assembly101: A large-scale multi-view video dataset for understanding procedural activities,” CVPR 2022

    2022

  14. [22]

    Fino-net: A deep multimodal sensor fusion framework for manipulation failure detection,

    A. Inceoglu, E. E. Aksoy, A. C. Ak, and S. Sariel, “Fino-net: A deep multimodal sensor fusion framework for manipulation failure detection,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 6841–6847

    2021

  15. [23]

    Conditionnet: Learning preconditions and ef- fects for execution monitoring,

    D. Sliwowski and D. Lee, “Conditionnet: Learning preconditions and ef- fects for execution monitoring,” IEEE Robotics and Automation Letters , 2024

    2024

  16. [24]

    Open x-embodiment: Robotic learning datasets and rt-x models : Open x-embodiment collaboration0,

    A. O’Neill et. al., “Open x-embodiment: Robotic learning datasets and rt-x models : Open x-embodiment collaboration0,” in IEEE International Conference on Robotics and Automation (ICRA) , 2024, pp. 6892–6903

    2024

  17. [25]

    On bringing robots home

    N. Muhammad, A. Rai, H. Etukuru, Y . Liu, I. Misra, S. Chintala, and L. Pinto, “On bringing robots home.” CoRR, 2023

    2023

  18. [26]

    Sparsh: Self-supervised touch representations for vision-based tactile sensing,

    C. Higuera et al., “Sparsh: Self-supervised touch representations for vision-based tactile sensing,” 2024

    2024

  19. [27]

    Learning to recognize objects in egocentric activities,

    A. Fathi, X. Ren, and J. M. Rehg, “Learning to recognize objects in egocentric activities,” in CVPR. IEEE, 2011, pp. 3281–3288

    2011

  20. [28]

    The language of actions: Recov- ering the syntax and semantics of goal-directed human activities,

    H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Recov- ering the syntax and semantics of goal-directed human activities,” in IEEE Conference on Computer Vision and Pattern Recognition , 2014, pp. 780–787

    2014

  21. [29]

    RT-1: Robotics Transformer for Real-World Control at Scale,

    A. Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale,” July 2023

    2023

  22. [30]

    M 3it: A large-scale dataset towards multi-modal multilin- gual instruction tuning,

    L. Li et al., “M 3it: A large-scale dataset towards multi-modal multilin- gual instruction tuning,” arXiv preprint arXiv:2306.04387 , 2023

    2023 arXiv

  23. [31]

    On-line rigid object recognition and pose estimation based on inertial parameters,

    D. Kubus, T. Kroger, and F. Wahl, “On-line rigid object recognition and pose estimation based on inertial parameters,” 12 2007, pp. 1402 – 1408

    2007

  24. [32]

    Low-latency automotive vision with event cameras,

    D. Gehrig and D. Scaramuzza, “Low-latency automotive vision with event cameras,” Nature, vol. 629, no. 8014, pp. 1034–1040, 2024

    2024

  25. [33]

    Combined physics and event camera simulator for slip detection,

    T. Reinold, S. Ghosh, and G. Gallego, “Combined physics and event camera simulator for slip detection,” in Proceedings of the Winter Conference on Applications of Computer Vision , 2025, pp. 935–943

    2025

  26. [34]

    Neuromorphic vision sensing for cnn-based action recognition,

    A . Chadha et al., “Neuromorphic vision sensing for cnn-based action recognition,” in IEEE ICASSP, 2019

    2019

  27. [35]

    Bilateral teleoperation with adaptive impedance control for contact tasks,

    Y . Michel, R. Rahal, C. Pacchierotti, P. R. Giordano, and D. Lee, “Bilateral teleoperation with adaptive impedance control for contact tasks,” IEEE Robotics and Automation Letters , vol. 6, no. 3, pp. 5429– 5436, 2021, publisher: IEEE

    2021

  28. [36]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  29. [37]

    Diffusion action segmentation,

    D. Liu, Q. Li, A.-D. Dinh, T. Jiang, M. Shah, and C. Xu, “Diffusion action segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 10 139–10 149

    2023

  30. [38]

    Transition state clustering: Unsupervised surgical tra- jectory segmentation for robot learning,

    S. Krishnan, A. Garg, S. Patil, C. Lea, G. Hager, P. Abbeel, and K. Goldberg, “Transition state clustering: Unsupervised surgical tra- jectory segmentation for robot learning,” The International journal of robotics research, vol. 36, no. 13-14, pp. 1595–1618, 2017

    2017

  31. [39]

    Multi-level task learning based on inten- tion and constraint inference for autonomous robotic manipulation,

    C. Willibald and D. Lee, “Multi-level task learning based on inten- tion and constraint inference for autonomous robotic manipulation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 7688–7695

    2022

  32. [40]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  33. [41]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems , vol. 34, pp. 8780–8794, 2021

    2021

  34. [42]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6299–6308

    2017

  35. [43]

    Raft: Recurrent all-pairs field transforms for optical flow,

    Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in ECCV 2020. Springer, 2020, pp. 402–419

    2020

  36. [44]

    M2R2: Mulitmodal robotic representation for temporal action segmentation,

    D. Sliwowski and D. Lee, “M2R2: Mulitmodal robotic representation for temporal action segmentation,” in Submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2025

    2025

  37. [45]

    Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,

    C. Chi et al., “Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots,” RSS, 2024

    2024

  38. [46]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” July 2023

    2023

  39. [47]

    Dynamic movement primitives in robotics: A tutorial survey,

    M. Saveriano, F. J. Abu-Dakka, A. Kramberger, and L. Peternel, “Dynamic movement primitives in robotics: A tutorial survey,” The International Journal of Robotics Research , vol. 42, no. 13, pp. 1133– 1184, 2023

    2023

  40. [48]

    Robotic peg-in-hole assembly based on reversible dynamic movement primitives and trajectory opti- mization,

    H. Zhao, Y . Chen, X. Li, and H. Ding, “Robotic peg-in-hole assembly based on reversible dynamic movement primitives and trajectory opti- mization,” Mechatronics, vol. 95, p. 103054, 2023

    2023

  41. [49]

    Search strategies for peg-in- hole assemblies with position uncertainty,

    S. R. Chhatpar and M. S. Branicky, “Search strategies for peg-in- hole assemblies with position uncertainty,” in IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No. 01CH37180) , vol. ...

    2001

  42. [50]

    A survey of behavior trees in robotics and ai,

    M. I. et al., “A survey of behavior trees in robotics and ai,” Robotics and Autonomous Systems , vol. 154, p. 104096, 2022. APPENDIX A DATA VISUALIZATION TOOL To simplify viewing the dataset, we designed a visualization tool based on the rerun Python package. rerun facilitates...

    2022

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.