Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Fusing RGB and point clouds via training-time modality dropout lets a visuomotor diffusion policy generalize robustly to scene changes, with a reported 39.1% average success-rate gain over unimodal baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:46 UTC pith:NCWFX2IB

load-bearing objection The paper's core insight—modality dropout matters more than fusion architecture—is supported by a clean ablation, but the headline 39.1% generalization gain is not backed by the reported data. the 5 major comments →

arxiv 2511.22445 v2 pith:NCWFX2IB submitted 2025-11-27 cs.RO

DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization

Yikai Tang , Haoran Geng , Jindou Jia , Yuxuan Hu , Sheng Zang , Jianfei Yang , Pieter Abbeel , Jitendra Malik This is my paper
classification cs.RO
keywords visuomotor policymultimodal fusionmodality dropoutdiffusion policypoint cloudRGBgeneralizationimitation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper (titled DIPOLE) proposes an approach it names Visual-Geometry Diffusion Policy (VGDP), a visuomotor diffusion policy that fuses an RGB image stream with a point-cloud stream during training by randomly dropping out one entire modality at each step before a lightweight cross-attention layer. The claim is that this simple training-time dropout — not the fusion architecture — enforces balanced, complementary use of the two modalities, so the learned representation stays informative when either one changes or degrades. Across 18 simulated and 4 real-world tasks, the authors report a 39.1% average success-rate improvement over six baselines, with 41.5% under visual randomization and 15.2% under spatial randomization. The strongest evidence cited is a task on which both unimodal encoders fail at or below 1% success while the fused policy reaches 72.3%, implying an emerging capability that neither modality alone provides. If these numbers hold, the work shows that a cheap regularizer can replace specialized fusion architectures for robust policy learning.

Core claim

The central discovery is that enforcing modality complementarity through random modality-wise dropout is the decisive ingredient for robust visuomotor generalization; bidirectional cross-attention merely provides a lightweight exchange mechanism. The authors show that without dropout, the fusion collapses to whichever unimodal stream is easier to fit at each randomization level, tracking the better baseline; with dropout, the fused policy maintains near-constant success across all three randomizations (2.53% relative dispersion) and exceeds even the best unimodal policy by roughly 11x in stability. This is not an architecture claim but a regularization claim about how to train multimodal pol

What carries the argument

The complementarity-aware fusion module: each training step masks out either the RGB branch or the point-cloud branch with probability 0.2, projects surviving features into a shared 256-dimensional space, passes them through a lightweight bidirectional cross-attention layer with residual connections, and applies element-wise dropout before the diffusion policy head. The modality-wise dropout is the load-bearing component: it forces each modality to remain individually informative, preventing the fused latent from collapsing onto a single dominant cue, while the cross-attention is shown in ablations to be useful but secondary.

Load-bearing premise

The load-bearing premise is that the reported 39.1% average improvement over baselines is computed on a complete and fair comparison — including the 12 simulated tasks whose per-task results are not shown and any seed-to-seed variance — so that the ordering of methods would survive a full audit.

What would settle it

Run the same 18-task benchmark with VGDP and the six baselines, reporting per-task success and per-seed variance for all tasks. If any baseline matches or beats VGDP on a majority of the unreported tasks, or if the aggregate advantage falls below the claimed 39.1% once the missing numbers are included, the central generalization claim fails. Alternatively, a single task from another independently maintained benchmark where a unimodal baseline beats VGDP under the same training budget would contradict the claim of general dominance.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported gains hold, policies trained with this dropout will maintain high success when one sensor is noisy or occluded, because each branch is trained to function alone.
  • The zero-shot results imply that a single policy trained this way can transfer to unseen objects, containers, and lighting without new demonstrations.
  • The near-flat performance across randomization levels (2.53% relative dispersion) suggests the representation is largely invariant to visual and spatial changes by construction, not by memorization.
  • The ablation pattern predicts that replacing cross-attention with cheaper fusion after dropout should retain most of the gains, guiding future efficient designs.
  • The 72.3% success on a task where both unimodal encoders fail at or below 1% indicates an emergent behavior: the fused representation enables skills neither individual modality supports.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same dropout-on-branches trick could plausibly generalize to other modality pairs (e.g., vision + tactile, or language + vision) where one modality tends to dominate; that extension is untested in the paper.
  • The paper's ablation suggests a testable hypothesis: modality-wise dropout alone, even with concatenative fusion, may outperform cross-attention without dropout; a direct comparison on a broader task set would isolate whether attention adds anything beyond the regularizer.
  • Because the real-world point-cloud baseline collapsed on noisy depth, the method may benefit from a learned depth denoiser or from applying dropout asymmetrically to the geometry branch; neither is explored.
  • A reader should probe the stability of the headline numbers by re-running the reported tasks with multiple seeds and publishing the distribution, since per-seed variance is not reported for any result.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper (whose metadata title is DIPOLE, although the full text uses VGDP) proposes a visuomotor diffusion policy that fuses a ResNet-18 RGB encoder with a DP3 point-cloud encoder through a complementarity-aware fusion layer. The core mechanism is training-time modality-wise dropout (p=0.2) that randomly disables one modality branch, followed by a lightweight bidirectional cross-attention layer, residual connections, and element-wise dropout. The authors claim that this combination yields stable high performance, robust visual and spatial generalization, sub-centimeter precision, emergent capabilities beyond either modality, and zero-shot transfer on 18 simulated and 4 real-world tasks, outperforming six to seven baselines by an average of 39.1%. An ablation study on the CloseBox task supports the claim that modality-wise dropout, rather than cross-attention alone, is the main driver of the gains.

Significance. If the headline empirical claims are correct, the paper makes a useful and simple contribution: a training-time regularizer that prevents modality collapse in RGB-point-cloud policy learning, with a widely applicable recipe for diffusion-policy observation encoders. The ablation study in Table V is an honest internal control and is the strongest part of the evidence. The comparison across a unified platform covering several simulators and real-world tasks is also potentially valuable. However, the significance of the paper currently rests on aggregated and partially reported evaluation results; the main empirical claims are not yet verifiable from the manuscript.

major comments (5)
  1. [IV.B, Tables II and III] The headline '39.1% average improvement' is not reproducible from the reported tables. Table II's caption quotes '40.11%' absolute improvement; Table III reports averages over tasks at three randomization levels but does not define whether 39.1% is an absolute percentage-point difference, a relative improvement, or an average of per-task relative gains. Recomputing from Table III gives roughly 40.7 percentage points absolute improvement, not 39.1%, and a much larger relative improvement. The paper should state the exact aggregation formula and report per-task and per-level numbers consistently.
  2. [IV.A, Table II] Only 6 of the 18 simulation tasks appear in Table II; the remaining 12 tasks are never individually reported. The bold claim that VGDP 'reaches 72.3% success even on tasks where ResNet and DP3 fail completely (≤1%)' is unsupported because the tasks underlying this number are not identified, and Table II does not contain any entry where both ResNet and DP3 are ≤1% and VGDP is near 0.72. The closest case, Tomato, has ResNet at 0.08, not ≤1%. Full per-task results for all 18 tasks are needed.
  3. [Table II and Table VI] The claim that 'VGDP prevails over all others on almost all tasks' is contradicted by the paper's own Table II: on StackCube, VGDP scores 0.03 while ResNet18-RGB scores 0.10. Similarly, in Table VI, VGDP's Fetch Bottle score (22.38%) is only marginally above DP(RGB) (18%). Without standard errors, confidence intervals, or results over multiple seeds, such close differences cannot be distinguished from noise. The paper states that each task is evaluated with 200 trials, but no variance or seed information is reported for any number. At minimum, provide means and standard deviations across at least three training seeds and report the number of evaluation trials per condition.
  4. [IV.A, 'Baseline Models'] The fairness and completeness of the comparison is not documented. The paper says all methods share 'the same diffusion-policy head,' but it does not report baseline hyperparameters: training steps, learning rates, batch sizes, image resolutions, point-cloud preprocessing, augmentation, or whether the baselines were reimplemented or taken from public checkpoints. Since the evaluation platform RoboVerse [19] is co-authored by this paper's authors, task selection and baseline configuration need to be transparent. Please release the full configuration files and per-task evaluation scripts, or provide them in the supplementary material.
  5. [V.A, V.D, Tables VI and VII] Real-world evaluation reporting is internally inconsistent. Section V.A says policies are 'evaluated on 25 trials, if not otherwise stated,' but Section V.B says FetchBottle is evaluated on all 143 grid positions, and Table VI reports 22.38%, which equals 32/143. The zero-shot transfer section states that VGDP achieves 'a perfect 8/9 zero-shot success rate,' but Table VII lists six transfer settings with a single checkmark per condition, and Figure 10 adds four spatial positions, giving a different denominator. Please reconcile the trial counts, define the denominators, and align the text with the tables.
minor comments (5)
  1. [Title/Abstract vs. Full Text] The paper is submitted as 'DIPOLE' in the abstract metadata but the full text consistently uses 'VGDP.' Please unify the name throughout.
  2. [IV.B] The term 'unimodal baselines' is used even though the comparison set includes RGBD ResNet, RGBD ViT, and MultiViT, which are multimodal encoders. Use 'baselines' or 'comparison methods' instead.
  3. [Table I] The 'Relative Dispersion' table lacks units and a clear formula. Define what 'relative dispersion' means (e.g., coefficient of variation) and how it is computed from the underlying success rates.
  4. [IV.D] The numbers '55.3% higher mean success rate' and '59.5% smaller IID–OOD gap' are presented without their derivation or a reference to the underlying table. Add the per-condition IID/OOD numbers or a figure with clear axes.
  5. [Section VII] The limitations paragraph is appropriately self-limiting, but it should also acknowledge that the 39.1% average is over a benchmark that is partly self-defined; the discussion of generalizability should be tempered accordingly.

Circularity Check

0 steps flagged

No significant circularity: VGDP is an empirical fusion method; no predicted quantity is defined by a fit or by a self-citation.

full rationale

This paper does not derive predictions from first principles in a way that could collapse into its inputs. The central claims are observed success rates after training on demonstrations and evaluating on held-out IID/OOD scenes. Equations (1)-(4) define a standard diffusion denoising loss conditioned on a fused context c; there is no equation in which a reported quantity is defined in terms of the target success rate or fitted to it. The 39.1%, 41.5%, 15.2%, and 72.3% figures are evaluation statistics, not fitted parameters renamed as predictions. The method components (modality-wise dropout, cross-attention) are training mechanisms, not quantities solved for from the benchmark outcomes. The paper does rely on RoboVerse [19] and Gello [43], which include overlapping authors, but these are cited as a public benchmarking platform and a teleoperation tool for data collection; they do not serve as an unverified theorem that entails the paper's conclusions. The limitations section (VII) explicitly narrows the benchmark scope, which is an evidence-completeness caveat rather than a circular step. Concerns about incomplete per-task reporting, missing variance information, and baseline hyperparameters are legitimate correctness/verifiability issues, but under the required standard—exhibiting a specific reduction such as Eq. X = Eq. Y by construction or a fitted input called a prediction—no circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

Empirical architecture paper: no new physics or formal derivation. The main axioms are design choices (dropout probability, embedding sizes, benchmark randomization levels) and the domain assumption that single-view depth back-projection yields usable metric geometry. Free parameters are hand-set or unstated (element-wise dropout rate); none are fitted to make a derivation work, but the central mechanism is sensitive to p, which is never swept.

free parameters (5)
  • modality dropout probability p = 0.2
    Hand-chosen; this is the paper's central mechanism, yet no sensitivity sweep is reported.
  • shared embedding dimension = 256
    Design choice for the fusion layer; never ablated.
  • point-cloud downsample count = 4096
    Farthest Point Sampling target; coverage-vs-compute choice, unablated.
  • DP3 point-cloud feature dimension = 64
    Inherited from the DP3 encoder design; VGDP depends on this small geometric feature.
  • element-wise dropout rate after fusion = not stated
    Used for stability but the rate is never given; a reproducibility gap.
axioms (4)
  • standard math The denoising diffusion objective (Eqs. 1–4) is an appropriate decision head.
    Follows Diffusion Policy [1]; the paper does not derive or modify the diffusion machinery.
  • domain assumption Single-view RGB-D back-projection yields a metric point cloud adequate for the 3D branch.
    §III.A: depth is back-projected and FPS-sampled. Table VI shows DP3 fails in real world (0–12%), so depth noise is load-bearing for VGDP's real-world numbers.
  • domain assumption RoboVerse randomization levels L0–L2 are a representative proxy for out-of-distribution generalization.
    §IV.A defines only three levels (materials/table, then camera); whether this matches real deployment shifts is assumed, not demonstrated.
  • ad hoc to paper Modality-wise dropout at p=0.2 enforces modality invariance rather than merely adding noise.
    §III.B: the central mechanism; ablated only on CloseBox (Table V), a single task.

pith-pipeline@v1.3.0-alltime-deepseek · 12274 in / 12134 out tokens · 109754 ms · 2026-08-03T19:46:01.146973+00:00 · methodology

0 comments
read the original abstract

Imitation learning has emerged as a crucial approach for acquiring visuomotor skills from demonstrations, where designing effective observation encoders is essential for policy generalization. However, existing methods tend to struggle once test-time conditions differ from the demonstrations, such as changes in lighting, texture, viewpoint, object placement, or object identity. To address this challenge, we propose DIffusion POlicy with compLementarity Encoders (DIPOLE), a visuomotor policy that learns to fuse complementary modalities through a training-time mechanism rather than a specialized fusion architecture. A modality-wise dropout masks one branch at each training step, encouraging each modality to remain individually informative. A lightweight cross-attention layer then exchanges complementary cues between the two. This design endows DIPOLE with five core strengths: stable high performance across diverse tasks, robustness to visual changes, spatial generalization at sub-centimeter precision, emergent capability beyond either modality, and zero-shot transfer to unseen objects. Across 18 simulated and 4 real-world tasks, DIPOLE outperforms six baselines by 39.1% on average, with gains of 41.5% under unseen visual distractors and 15.2% under randomized object placement.

Figures

Figures reproduced from arXiv: 2511.22445 by Haoran Geng, Jianfei Yang, Jindou Jia, Jitendra Malik, Pieter Abbeel, Sheng Zang, Yikai Tang, Yuxuan Hu.

Figure 1
Figure 1. Figure 1: Visual-Geometry Diffusion Policy (VGDP) is an imitation learning method that fuses 3D observations with 2D images through a Complementarity-Aware Fusion Module, which uses modality-wise dropout to enforce balanced use of RGB and geometry. This design yields substantial improvements in average performance, generalization, and robustness. VGDP is extensively evaluated in both simulation and the real world, c… view at source ↗
Figure 2
Figure 2. Figure 2: Visual-Geometry Diffusion Policy Overview. (a) Observation: The environment is captured by a single-view RGB-D camera together with robot joint states. (b) Perception: Each modality is independently encoded by a dedicated encoder, ensuring comprehensive semantic representation. (c) Integration: The richly represented features learn cross-modal dependencies and contextual relationships via a cross-attention… view at source ↗
Figure 3
Figure 3. Figure 3: Comparative Analysis of Encoder Performance. Figure 3a shows how different encoders perform across tasks, with each task score averaged over the three randomization levels. Figure 3b depicts their performance across randomization levels, where each level score is averaged over all tasks. Relative Dispersion VGDP RGB resnet18 RGBD resnet18 RGBD ViT RGBD MultiViT PointCloud DP3 PointCloud spUnet Cross-task ↓… view at source ↗
Figure 4
Figure 4. Figure 4: Environments under Different Randomization Levels. The three brackets, named L0, L1 and L2, demon￾strate example randomization in materials, positions and viewpoints of the same task across different randomization levels. Level materials table position camera pos L0 ✗ ✗ ✗ L1 ✓ ✓ ✗ L2 ✓ ✓ ✓ TABLE IV: Randomization factors at different levels. Level 1 randomizes wall, table, and object materials and rotates … view at source ↗
Figure 5
Figure 5. Figure 5: Real-world Benchmarks. We deploy VGDP in the real world with a Franka Arm across four challenging tasks, including clutter scene manipulation (PickButter), 6-DoF cereal pouring (PourCereal), fine-grained handling over large spatial generalization (FetchBottle) and precision-driven, force-aware insertion (InsertPlug). (0.73/0.77), and the fused model again shifts toward the point-cloud branch (0.70/0.67). A… view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation Results for Real-world Randomization. The grid map represents a bird’s-eye-view of the workspace, where each grid corresponds to a bottle placement during evaluation. Policies are trained on expert demonstrations on the 30 dotted grids and evaluated on all 143 positions. Dark￾green, light-green, and white represent IID success, OOD success, and failure, respectively. Practical imitation learning… view at source ↗
Figure 8
Figure 8. Figure 8: Evaluation Results for Insert Plug. The grid map represents a bird’s-eye-view of the workspace, where each grid represents a position where the plug is placed during evaluation; policies are trained on expert demonstrations on all dotted grids and evaluated on selected 25 positions that spans evenly across the workspace; grids colored in dark￾green and white represent success and failure, respectively. D. … view at source ↗
Figure 9
Figure 9. Figure 9: Zero-shot transfer settings. Illustration of the trans￾fer conditions used in our real-world evaluation. (Up): In Pour Cereal, we replace the plate with containers of various size and shape including: a drawer, a bowl and a yogurt cup. (Down Left): In Insert Plug, we either replace the IPad with an airpods case or move the IPad to a new position. (Down Right) In Fetch Bottle, we replace the original lighti… view at source ↗
Figure 10
Figure 10. Figure 10: Spatial zero-shot transfer in FetchBottle. VGDP succeeds on three of four OOD positions, while unimodal baselines fail consistently. Unified Analysis. Together, these six transfer settings span a broad spectrum of real-world distribution shifts: (1) seman￾tic (new containers, new devices), (2) geometric (different receptacle shapes and insertion hardware), (3) visual (harsh RGB lighting changes), and (4) … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions

    cs.LG 2026-02 conditional novelty 4.0

    Behavior-cloning pretraining from AR demonstrations plus a contrastive projection-head loss in SAC improves dexterous grasping success by ~8 points and cuts training time ~4x in simulation.

Reference graph

Works this paper leans on

51 extracted references · 40 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” 2024. [Online]. Available: https://arxiv.org/abs/2303.04137

    Pith/arXiv arXiv 2024

  2. [2]

    Action chunking with transformers for image- based spacecraft guidance and control,

    A. Posadas-Nava, A. Scorsoglio, L. Ghilardi, R. Furfaro, and R. Linares, “Action chunking with transformers for image- based spacecraft guidance and control,” 2025. [Online]. Available: https://arxiv.org/abs/2509.04628

    Pith/arXiv arXiv 2025

  3. [3]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations,

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations,” 2024. [Online]. Available: https://arxiv.org/abs/2403.03954

    Pith/arXiv arXiv 2024

  4. [4]

    Discrete policy: Learning disentangled action space for multi-task robotic manipulation,

    K. Wu, Y . Zhu, J. Li, J. Wen, N. Liu, Z. Xu, and J. Tang, “Discrete policy: Learning disentangled action space for multi-task robotic manipulation,” 2025. [Online]. Available: https://arxiv.org/abs/2409.18707

    Pith/arXiv arXiv 2025

  5. [5]

    Perceiver-actor: A multi-task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” 2022. [Online]. Available: https://arxiv.org/abs/2209.05451

    Pith/arXiv arXiv 2022

  6. [6]

    Learning 3d dynamic scene representations for robot manipulation,

    Z. Xu, Z. He, J. Wu, and S. Song, “Learning 3d dynamic scene representations for robot manipulation,” 2020. [Online]. Available: https://arxiv.org/abs/2011.01968

    Pith/arXiv arXiv 2020

  7. [7]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J...

    Pith/arXiv arXiv 2023

  8. [8]

    Gp3: A 3d geometry-aware policy with multi- view images for robotic manipulation,

    Q. Qian, G. Zhao, G. Zhang, J. Wang, R. Xu, J. Gao, and D. Zhao, “Gp3: A 3d geometry-aware policy with multi- view images for robotic manipulation,” 2025. [Online]. Available: https://arxiv.org/abs/2509.15733

    arXiv 2025

  9. [9]

    Stem-ob: Generalizable visual imitation learning with stem-like convergent observation through diffusion inversion,

    K. Hu, Z. Rui, Y . He, Y . Liu, P. Hua, and H. Xu, “Stem-ob: Generalizable visual imitation learning with stem-like convergent observation through diffusion inversion,” 2024. [Online]. Available: https://arxiv.org/abs/2411.04919

    Pith/arXiv arXiv 2024

  10. [10]

    Rise: 3d perception makes real-world robot imitation simple and effective,

    C. Wang, H. Fang, H.-S. Fang, and C. Lu, “Rise: 3d perception makes real-world robot imitation simple and effective,” 2024. [Online]. Available: https://arxiv.org/abs/2404.12281

    Pith/arXiv arXiv 2024

  11. [11]

    Zero-shot visual general- ization in robot manipulation,

    S. Batra and G. Sukhatme, “Zero-shot visual general- ization in robot manipulation,” 2025. [Online]. Available: https://arxiv.org/abs/2505.11719

    Pith/arXiv arXiv 2025

  12. [12]

    Deformable detr: Deformable transformers for end-to-end object detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” 2021. [Online]. Available: https://arxiv.org/abs/2010.04159

    Pith/arXiv arXiv 2021

  13. [13]

    Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot,

    H.-S. Fang, H. Fang, Z. Tang, J. Liu, C. Wang, J. Wang, H. Zhu, and C. Lu, “Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot,” 2023. [Online]. Available: https://arxiv.org/abs/2307.00595

    Pith/arXiv arXiv 2023

  14. [14]

    Pointfusion: Deep sensor fusion for 3d bounding box estimation,

    D. Xu, D. Anguelov, and A. Jain, “Pointfusion: Deep sensor fusion for 3d bounding box estimation,” 2018. [Online]. Available: https://arxiv.org/abs/1711.10871

    Pith/arXiv arXiv 2018

  15. [15]

    Mvx-net: Multimodal voxelnet for 3d object detection,

    V . A. Sindagi, Y . Zhou, and O. Tuzel, “Mvx-net: Multimodal voxelnet for 3d object detection,” 2019. [Online]. Available: https://arxiv.org/abs/1904.01649

    Pith/arXiv arXiv 2019

  16. [16]

    Pointpainting: Sequential fusion for 3d object detection,

    S. V ora, A. H. Lang, B. Helou, and O. Beijbom, “Pointpainting: Sequential fusion for 3d object detection,” 2020. [Online]. Available: https://arxiv.org/abs/1911.10150

    Pith/arXiv arXiv 2020

  17. [17]

    Pointaugment: an auto-augmentation framework for point cloud classification,

    R. Li, X. Li, P.-A. Heng, and C.-W. Fu, “Pointaugment: an auto-augmentation framework for point cloud classification,” 2020. [Online]. Available: https://arxiv.org/abs/2002.10876

    Pith/arXiv arXiv 2020

  18. [18]

    Towards fusing point cloud and visual representations for imitation learning,

    A. Donat, X. Jia, X. Huang, A. Taranovic, D. Blessing, G. Li, H. Zhou, H. Zhang, R. Lioutikov, and G. Neumann, “Towards fusing point cloud and visual representations for imitation learning,” 2025. [Online]. Available: https://arxiv.org/abs/2502.12320

    Pith/arXiv arXiv 2025

  19. [19]

    Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning,

    H. Geng, F. Wang, S. Wei, Y . Li, B. Wang, B. An, C. T. Cheng, H. Lou, P. Li, Y .-J. Wang, Y . Liang, D. Goetting, C. Xu, H. Chen, Y . Qian, Y . Geng, J. Mao, W. Wan, M. Zhang, J. Lyu, S. Zhao, J. Zhang, J. Zhang, C. Zhao, H. Lu, Y . Ding, R. Gong, Y . Wang, Y . Kuang, R. Wu, B. Jia, C. Sferrazza, H. Dong, S. Huang, Y . Wang, J. Malik, and P. Abbeel, “Rob...

  20. [20]

    Interactive imitation learning in robotics: A survey,

    C. Celemin, R. P ´erez-Dattari, E. Chisari, G. Franzese, L. de Souza Rosa, R. Prakash, Z. Ajanovi ´c, M. Ferraz, A. Valada, and J. Kober, “Interactive imitation learning in robotics: A survey,”

  21. [21]

    Bridgedata v2: A dataset for robot learning at scale,

    H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V . Myers, K. Fang, C. Finn, and S. Levine, “Bridgedata v2: A dataset for robot learning at scale,”

  22. [22]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space,

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” 2017. [Online]. Available: https://arxiv.org/abs/1706.02413

    Pith/arXiv arXiv 2017

  23. [23]

    Minkloc++: Lidar and monocular image fusion for place recognition,

    J. Komorowski, M. Wysoczanska, and T. Trzcinski, “Minkloc++: Lidar and monocular image fusion for place recognition,” 2021. [Online]. Available: https://arxiv.org/abs/2104.05327

    Pith/arXiv arXiv 2021

  24. [24]

    Minkloc3d: Point cloud based large- scale place recognition,

    J. Komorowski, “Minkloc3d: Point cloud based large- scale place recognition,” 2020. [Online]. Available: https://arxiv.org/abs/2011.04530

    Pith/arXiv arXiv 2020

  25. [25]

    Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection,

    S. Xu, D. Zhou, J. Fang, J. Yin, Z. Bin, and L. Zhang, “Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection,”

  26. [26]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015. [Online]. Available: https://arxiv.org/abs/1512.03385

    Pith/arXiv arXiv 2015

  27. [27]

    Adapt3r: Adaptive 3d scene representation for domain transfer in imitation learning,

    A. Wilcox, M. Ghanem, M. Moghani, P. Barroso, B. Joffe, and A. Garg, “Adapt3r: Adaptive 3d scene representation for domain transfer in imitation learning,” 2025. [Online]. Available: https://arxiv.org/abs/2503.04877

    Pith/arXiv arXiv 2025

  28. [28]

    Cordvip: Correspondence-based visuomotor policy for dexterous manipulation in real-world,

    Y . Fu, Q. Feng, N. Chen, Z. Zhou, M. Liu, M. Wu, T. Chen, S. Rong, J. Liu, H. Dong, and S. Zhang, “Cordvip: Correspondence-based visuomotor policy for dexterous manipulation in real-world,” 2025. [Online]. Available: https://arxiv.org/abs/2502.08449

    Pith/arXiv arXiv 2025

  29. [29]

    Polarnet: 3d point clouds for language-guided robotic manipulation,

    S. Chen, R. Garcia, C. Schmid, and I. Laptev, “Polarnet: 3d point clouds for language-guided robotic manipulation,” 2023. [Online]. Available: https://arxiv.org/abs/2309.15596

    Pith/arXiv arXiv 2023

  30. [30]

    V oxelnet: End-to-end learning for point cloud based 3d object detection,

    Y . Zhou and O. Tuzel, “V oxelnet: End-to-end learning for point cloud based 3d object detection,” 2017. [Online]. Available: https://arxiv.org/abs/1711.06396

    Pith/arXiv arXiv 2017

  31. [31]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2003.08934

    Pith/arXiv arXiv 2020

  32. [32]

    3d gaussian splatting for real-time radiance field rendering,

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” 2023. [Online]. Available: https://arxiv.org/abs/2308.04079

    Pith/arXiv arXiv 2023

  33. [33]

    Robust multimodal learning via representation decoupling,

    S. Wei, Y . Luo, Y . Wang, and C. Luo, “Robust multimodal learning via representation decoupling,” 2024. [Online]. Available: https://arxiv.org/abs/2407.04458

    Pith/arXiv arXiv 2024

  34. [34]

    Robust multimodal learning with missing modalities via parameter-efficient adaptation,

    M. K. Reza, A. Prater-Bennette, and M. S. Asif, “Robust multimodal learning with missing modalities via parameter-efficient adaptation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 2, p. 742–754, Feb. 2025. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2024.3476487

    arXiv 2025

  35. [35]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  36. [36]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2021

  37. [37]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” inInternational Confer- ence on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241

    2015

  38. [38]

    Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,”Advances in Neural Information Processing Systems, vol. 36, pp. 44 776–44 791, 2023

    2023

  39. [39]

    Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations,

    T. Mu, Z. Ling, F. Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su, “Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations,” 2021. [Online]. Available: https://arxiv.org/abs/2107.14483

    Pith/arXiv arXiv 2021

  40. [40]

    Maniskill2: A unified benchmark for generalizable manipulation skills,

    J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y . Tang, S. Tao, X. Wei, Y . Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su, “Maniskill2: A unified benchmark for generalizable manipulation skills,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04659

    Pith/arXiv arXiv 2023

  41. [41]

    Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,

    S. Tao, F. Xiang, A. Shukla, Y . Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y . Liu, T. kai Chan, Y . Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V . N. Rajesh, Y . W. Choi, Y .-R. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su, “Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,” 2025. [Online]. Available: ...

    Pith/arXiv arXiv 2025

  42. [42]

    Rlbench: The robot learning benchmark & learning environment,

    S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3019–3026, 2020

    2020

  43. [43]

    Gello: A general, low- cost, and intuitive teleoperation framework for robot manipulators,

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel, “Gello: A general, low- cost, and intuitive teleoperation framework for robot manipulators,”

  44. [44]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” 2022. [Online]. Available: https://arxiv.org/abs/2202.02005

    Pith/arXiv arXiv 2022

  45. [45]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,

    T. Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” 2021. [Online]. Available: https://arxiv.org/abs/1910.10897

    Pith/arXiv arXiv 2021

  46. [46]

    Learning fine-grained bimanual manipulation with low-cost hardware,

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” 2023. [Online]. Available: https://arxiv.org/abs/2304.13705

    Pith/arXiv arXiv 2023

  47. [48]

    Available: https://arxiv.org/abs/2309.13037

    [Online]. Available: https://arxiv.org/abs/2309.13037

    Pith/arXiv arXiv

  48. [2021]

    Available: https://arxiv.org/abs/2106.12449

    [Online]. Available: https://arxiv.org/abs/2106.12449

    Pith/arXiv arXiv

  49. [2022]

    Available: https://arxiv.org/abs/2211.00600

    [Online]. Available: https://arxiv.org/abs/2211.00600

    Pith/arXiv arXiv

  50. [2024]

    Available: https://arxiv.org/abs/2308.12952

    [Online]. Available: https://arxiv.org/abs/2308.12952

    Pith/arXiv arXiv

  51. [2025]

    Available: https://arxiv.org/abs/2504.18904

    [Online]. Available: https://arxiv.org/abs/2504.18904

    Pith/arXiv arXiv