Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Spatial-Temporal Aware Visuomotor Diffusion Policy Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that training a diffusion visuomotor policy to reconstruct the current scene and predict the next scene through a dynamic Gaussian world model raises success rates by 16.4 percentage points on Adroit, 14.0 on DexArt, 6.45…

desk verdict A plausible but incremental extension of ManiGaussian: consistent gains across many benchmarks, but the spatial-temporal awareness story is not supported, the ablation text conflicts with Table 4, and a key baseline is missing. read the letter →

arxiv 2507.06710 v2 pith:ZHPTHM4E submitted 2025-07-09 cs.RO

Zhenyang Liu , Yikai Wang , Kuanning Wang , Longfei Liang , Xiangyang Xue , Yanwei Fu This is my paper
classification cs.RO
keywords visualimitationlearningdiffusionpolicyGaussiansplattingworldmodelrobotmanipulation3Dspatialawareness4DspatiotemporaldynamicsRGB-Dobservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that imitation learning from recorded trajectories misses the 3D structure of scenes and the 4D dynamics of interaction, and that this gap can be closed with auxiliary supervision rather than new hardware. To that end it introduces DP4, a diffusion policy whose representation is trained by reconstructing the current single-view RGB-D scene as 3D Gaussians and by deforming those Gaussians under the candidate action to render the predicted next scene. The two extra losses—$L_{3D}$ for current-scene reconstruction and $L_{4D}$ for future-scene prediction—are added to the action-diffusion objective. Across 17 simulated tasks with 173 variants and 3 real-robot tasks, the paper reports higher average success than 2D and 3D diffusion baselines, including +16.4% on Adroit, +14.0% on DexArt, +6.45% on RLBench, and +8.6% on real robots. If true, this is a practical route to spatial-temporal awareness in visuomotor policies because the world model is used only during training and adds no inference cost.

What carries the argument

The engine of the method is the dynamic Gaussian world model: a scene is represented as a set of 3D Gaussian primitives (positions, colors, rotations, scales, opacities) rendered into images by alpha-blending, and a deformable MLP predicts how those primitives change under an action to render the next frame. It does three jobs at once: it supervises the 3D global representation through rendered RGB and depth, it supervises temporal dynamics through the future-frame rendering loss, and it conditions the diffusion policy's trajectory generation on the resulting multi-level 3D features. Because the world model is discarded at inference, the claimed benefit is representation quality rather than extra computation.

What would settle it

Train DP4 with the depth channel stripped from $L_{3D}$ while keeping the RGB reconstruction loss; if success rates on Adroit stay near 84.7%, then the 3D spatial mechanism is not what carries the gain. Alternatively, replace the ground-truth future frame in $L_{4D}$ with a randomly chosen future frame from another episode; if performance does not drop, the temporal consistency term is not the cause.

Watch

Extended reading notes

Core claim

The central discovery is that a visuomotor diffusion policy can be made spatially and temporally aware by supervising a dynamic Gaussian world model learned alongside the policy, without changing how the policy acts at test time. From one RGB-D image the model builds a voxel scene representation, a generalizable Gaussian regressor turns it into 3D Gaussian primitives, and a rendering loss enforces agreement with the observed color and depth. A deformable mapper then shifts the Gaussian parameters using the current action, renders a predicted next RGB-D frame, and a second loss enforces agreement with the real future frame. The full objective is $L_{DP4} = L_{action} + \lambda_{3D} L_{3D} + \lambda_{4D} L_{4D}$, and the paper reports that the added terms improve success rates on all evaluated benchmarks while leaving inference time essentially unchanged.

Load-bearing premise

The load-bearing premise is that the single-view rendering losses teach genuine 3D geometry and temporal dynamics, since the 3D loss compares a rendered image with the same view that produced it and the 4D loss checks only one step ahead, so the model could in principle memorize appearance without building consistent structure.

Editorial extensions

If this is right

  • Point-cloud diffusion policies gain consistent success improvements from 3D and 4D rendering losses across Adroit, DexArt, and RLBench, with the largest relative gains on dexterous and articulated manipulation.
  • In the ablations, the 4D future-scene term is the largest single lever, lifting Pen success from 47% to 75%.
  • A single RGB-D camera with known intrinsics and extrinsics suffices for both training supervision and deployment, so the method does not require a multi-camera rig.
  • Because the world model is training-only, the policy's inference cost stays close to the point-cloud diffusion baseline, making the added supervision a low-cost upgrade for existing visuomotor policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension, not made in the paper, is to attach the same two rendering losses to non-diffusion visuomotor policies, since the loss structure is not diffusion-specific.
  • The paper evaluates only one-step future prediction, so a natural next test is a multi-step rollout of the deformable mapper; long-horizon tasks such as pouring would reveal whether the temporal model generalizes beyond a single step.
  • The reported gains could in principle come from regularization rather than geometric understanding; ablating depth from $L_{3D}$ or shuffling future frames in $L_{4D}$ would separate those explanations.
  • Because the world model is trained on the same view used by the policy, a further test of genuine 3D structure would be to render from a held-out camera pose and measure reconstruction error there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes 4D Diffusion Policy (DP4), a visuomotor diffusion policy augmented with two auxiliary scene-level losses: a 3D loss that reconstructs the current RGB-D observation via Gaussian splatting from a single view, and a 4D loss that predicts the next RGB-D frame from the current observation and action through a deformable Gaussian world model. The authors report consistent improvement over existing visuomotor policies across Adroit, DexArt, and RLBench simulation benchmarks (aggregate gains of +16.4%, +14.0%, and +6.45%) and across three real-robot tasks (+8.6% on average). The Gaussian world model is used only during training; at inference the policy relies on the multi-level 3D representation. The central claim is that these losses instill genuine 3D spatial and 4D spatiotemporal awareness in the representation, which in turn improves trajectory generation.

Significance. If the mechanism claim is correct, DP4 is a meaningful step toward explicit scene-level spatial-temporal supervision in visuomotor policy learning: it is one of relatively few approaches that couple a diffusion policy with a dynamic Gaussian world model, and the experimental scope (17 simulated tasks plus real-robot tasks) is appropriate for a policy-learning paper. The paper also provides architecture details and PyTorch snippets for the main components, which aids reproducibility. However, the significance is currently conditional: the paper does not show that the auxiliary losses actually produce geometric or temporal understanding rather than serving as generic regularizers, and the evaluation and ablation contain inconsistencies that weaken the quantitative claims. These issues are addressable, but they need to be fixed before the paper can be accepted.

major comments (4)
  1. [§3.2, Eq. (2)] The 3D spatial supervision is a same-view reconstruction loss: L3D compares a rendered RGB-D image to the exact input view from which the point cloud was created. Because the depth image already provides 3D coordinates, this loss can in principle be minimized by per-point appearance fitting without any multi-view or cross-geometry consistency. The paper calls this '3D spatial awareness' but provides no novel-view rendering evaluation or any other evidence that this loss enforces consistent 3D structure. The authors even concede in §4.3 that the renderings 'lack fine detail.' I recommend adding a quantitative novel-view or held-out-view rendering evaluation, or an ablation that removes depth supervision to separate geometry from appearance, to support the mechanism claim.
  2. [§3.3, Eq. (4) and §4.4] The 4D loss is a one-step future-frame reconstruction computed only during training, and §4.4 states that the Gaussian world model is not constructed at inference. Therefore any benefit of L4D must transfer to the shared multi-level 3D representation. The paper does not provide evidence that L4D contributes temporal structure rather than acting as an extra regularizer. Concretely, I would like to see a control experiment where the future target in Eq. (4) is replaced by the current frame or by an action-independent random future frame, or a representation-probing experiment that checks whether the learned features encode future state information. Without such a control, the central '4D spatiotemporal awareness' interpretation is unsupported.
  3. [§4.4, Table 4 and text] The component ablation contains explicit numerical inconsistencies between the table and the surrounding text. For the Door task, the text reports the base model at 94.0% and DP4 at 100.0%, whereas Table 4 reports 64.0% and 80.0%, respectively. For the Hammer task, the text says DP4 reaches 98.0% while the table shows 100.0%. Since this table is the primary evidence for component attribution (3D loss, 4D loss, and their combination), these inconsistencies make the attribution unreliable. The authors should correct the numbers and state clearly which result is final.
  4. [§4.1 evaluation metric and Table 3] The evaluation protocol for Adroit and DexArt uses the mean of the top five success rates across checkpoints, and the paper reports no significance tests for any benchmark. Selecting the best five checkpoints out of the evaluation history can systematically inflate reported performance and can inflate the measured gap between methods, especially with only three seeds. In addition, Table 3 (RLBench) omits DP3, which is identified in §4.1 as the primary baseline of the paper. Since DP3 is the closest point-cloud-based comparison and the one most likely to isolate the effect of the proposed auxiliary losses, its absence is a significant gap. I recommend reporting fixed-checkpoint performance with confidence intervals and adding DP3 to the RLBench comparison.
minor comments (5)
  1. [Abstract and §4.1] The abstract states '17 simulation tasks with 173 variants' and the introduction mentions 'four benchmarks,' but Section 4.1 only lists Adroit, DexArt, and RLBench and says 'we gathered 22 tasks.' Please reconcile these counts.
  2. [§3.3, Eq. (3)] The text states that 'w denotes the camera pose used to project the Gaussian primitives,' but the camera pose does not appear in Eq. (3); the rendering is written as R(θ(t+1), p). Either include w in the notation or remove the undefined symbol.
  3. [Figure 5] The caption and axis labels are garbled ('T+3 T+6 T+9T'). Please redraw the figure with clear ordering of the predicted future frames.
  4. [§4.5 and Table 5] The real-robot evaluation would benefit from a statement of the number of trials and whether success rates are averaged over the same initial conditions; currently only percentages are reported without error bars.
  5. [§4.1] Typo: 'KINOV A GEN2' should be 'Kinova Gen2'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 3D/4D auxiliary losses are training-only supervision signals, and the success-rate improvements are empirical claims that do not reduce to the objectives by construction.

full rationale

The paper's derivation chain is not circular. The auxiliary objectives L3D (Eq. 2) and L4D (Eq. 4) are self-supervised reconstruction and one-step future-frame prediction losses whose targets are ground-truth RGB-D observations from the demonstration dataset; they are additional training regularizers added to the diffusion action loss in Eq. (7), not quantities that are renamed as predictions of task success. The Gaussian world model is explicitly discarded at inference (Sec. 4.4), so no inference-time output is defined in terms of the reported success metric. The paper contains no load-bearing self-citations: it reuses DP3's official backbone and cites standard 3DGS/dynamic-Gaussian references for the rendering machinery. The paper even concedes (Sec. 4.3) that single-view renderings 'lack fine detail,' which weakens the spatial-awareness mechanism claim but is a limitation, not a circularity. The internal inconsistency between Sec. 4.4's Door/Hammer numbers and Table 4's entries, the top-five-checkpoint evaluation metric, and the omission of DP3 from RLBench comparisons are evaluation-robustness and correctness risks rather than input-output equivalences. Since no prediction or first-principles result reduces to a fitted parameter or to a self-citation, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard ML machinery (diffusion, 3DGS) plus the modeling assumptions that Gaussian parameter shifts can represent scene dynamics and that single-view rendering losses provide true spatial-temporal supervision. The only fitted hyperparameters are the two loss weights, tuned on one task.

free parameters (2)
  • lambda_3D = 0.1
    Chosen by hyperparameter sweep on the Adroit Pen task (Section 4.4, Figure 6); used for all tasks and domains.
  • lambda_4D = 0.01
    Same sweep as lambda_3D; used for all tasks and domains.
assumptions (5)
  • domain assumption 3D Gaussian Splatting differentiable rasterization is an appropriate generative model for scene geometry from point clouds.
    Invoked in Section 3.2 to construct the Gaussian world model; the paper relies on the renderer's gradients to supervise the encoder.
  • domain assumption The Gaussian primitive parameters θ(t) plus action a(t) are sufficient to predict the next scene's appearance, i.e., the deformable MLP p_phi in Eq. (3) can represent environment dynamics.
    Section 3.3, Eq. (3); if one-step dynamics are not expressible in Gaussian parameter shifts, L4D cannot provide the claimed temporal supervision.
  • domain assumption A single-view RGB-D observation with known camera intrinsics/extrinsics is sufficient to construct a useful 3D representation for manipulation.
    Sections 3.1-3.2; the entire pipeline avoids multi-view setups.
  • domain assumption Expert demonstrations provide ground truth future RGB-D observations (C*(t+1), D*(t+1)) for the 4D loss.
    Section 3.3, Eq. (4); in offline imitation settings this is plausible but still an assumption about dataset availability.
  • domain assumption The top-five success-rate selection over training checkpoints is a valid estimator of policy performance.
    Section 4.1 evaluation protocol; this is a methodological assumption that is not justified or cited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spatial-Temporal Aware Visuomotor Diffusion Policy Learning." pith.science (2026). https://pith.science/paper/ZHPTHM4E

@misc{pith2026250706710,
  author       = {Pith},
  title        = {Pith review of: Spatial-Temporal Aware Visuomotor Diffusion Policy Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHPTHM4E}},
  note         = {Machine review of arXiv:2507.06710}
}
read the original abstract

Visual imitation learning is effective for robots to learn versatile tasks. However, many existing methods rely on behavior cloning with supervised historical trajectories, limiting their 3D spatial and 4D spatiotemporal awareness. Consequently, these methods struggle to capture the 3D structures and 4D spatiotemporal relationships necessary for real-world deployment. In this work, we propose 4D Diffusion Policy (DP4), a novel visual imitation learning method that incorporates spatiotemporal awareness into diffusion-based policies. Unlike traditional approaches that rely on trajectory cloning, DP4 leverages a dynamic Gaussian world model to guide the learning of 3D spatial and 4D spatiotemporal perceptions from interactive environments. Our method constructs the current 3D scene from a single-view RGB-D observation and predicts the future 3D scene, optimizing trajectory generation by explicitly modeling both spatial and temporal dependencies. Extensive experiments across 17 simulation tasks with 173 variants and 3 real-world robotic tasks demonstrate that the 4D Diffusion Policy (DP4) outperforms baseline methods, improving the average simulation task success rate by 16.4% (Adroit), 14% (DexArt), and 6.45% (RLBench), and the average real-world robotic task success rate by 8.6%.

Figures

Figures reproduced from arXiv: 2507.06710 by the authors.

Figure 1
Figure 1. Spatial-temporal awareness in the 4D Diffusion Policy (DP4). Previous methods train perception and decision￾making with trajectory supervision, but trajectory cloning fails to capture the 3D spatial and 4D spatiotemporal relationships. In contrast, DP4 constructs the current 3D scene with 3D spatial su￾pervision from a single RGB-D view and predicts future 3D scene candidates using 4D spatiotemporal supervision, opt… view at source ↗
Figure 2
Figure 2. The framework of our 4D Diffusion Policy (DP4). From a single-view RGB-D observation, we construct 3D point clouds and extract global and local features to enrich both holistic and focused perceptions. These multi-level representations condition the diffusion policy model to generate trajectories based on current robot states. We introduce a Gaussian world model in DP4 to capture 3D structures and 4D spatiotemporal … view at source ↗
Figure 4
Figure 4. Visualization of Gaussian world model. After training DP4, we visualize its Gaussian world model by rendering a single￾view observation. DP4 uses a Gaussian world model to provide supervision, facilitating the learning of 3D spatial perception. 94.0% success rate and a completion time of 6.40 sec￾onds. After adding the Gaussian world model with RGB supervision, success improved to 96.0%, with a minimal increase in t… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Qualitative case study. The red mark indicates a pose that significantly deviates from the expert demonstration, while the green mark denotes a pose that aligns with the expert trajec￾tory. The 4D Diffusion Policy (DP4) integrates 3D spatial and 4D spatiotemporal aware…
Figure 5
Figure 5. Figure 5: More visualizations of the learned world model. DP4 uses a Gaussian world model to provide supervision, facilitating the 4D spatiotemporal perceptions in interactive environments. Results on Adroit Simulation 3D Component 4D Component Performance RGB Depth Dynamics Ham…
Figure 7
Figure 7. Figure 7: Visualization of DP4 performance on three real￾world robotic tasks. DP4 demonstrates strong performance in real-world settings and effectively handles a variety of common tasks with a single view. and Pouring Water. We focus on the model’s performance at key frames, wh…
Figure 6
Figure 6. Figure 6: Hyperparameter analysis of DP4. The effects of λ3D and λ4D on success rates, respectively. Method Grasping Bottles Stacking Cups Pouring Water Overall DP [5] 36.0 44.0 28.0 36.0 DP3 [45] 42.0 62.0 34.0 46.0 DP4 (ours) 48.0 72.0 44.0 54.6 [PITH_FULL_IMAGE:figures/full_…
Figure 8
Figure 8. Figure 8: Keyframes for real robot tasks. We give the keyframes used in our 3 real robot tasks. interactions involve planar friction. Object-fingertip inter￾actions support torsion and rolling friction. DexArt. This simulation task includes four dexterous ma￾nipulation tasks: Fa…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    LifelongVLA pairs dual-timescale LoRA gating with stochastic cached-prefix replay to cut catastrophic forgetting in VLA policies, reporting 83.2% average success and 11.4% forgetting on a 10-task LIBERO stream.

Reference graph

Works this paper leans on

50 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Dexterous imitation made easy: A learning- based framework for efficient dexterous manipulation

    Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto. Dexterous imitation made easy: A learning- based framework for efficient dexterous manipulation. In 2023 ieee international conference on robotics and automa- tion (icra), pages 5954–5961. IEEE, 2023. 2

    work page 2023

  2. [2]

    Dexart: Benchmarking generalizable dexterous manipu- lation with articulated objects

    Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipu- lation with articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21190–21200, 2023. 5

    work page 2023

  3. [3]

    Decision transformer: Reinforce- ment learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srini- vas, and Igor Mordatch. Decision transformer: Reinforce- ment learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021. 1

    work page 2021

  4. [4]

    Dif- fusiondet: Diffusion model for object detection

    Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Dif- fusiondet: Diffusion model for object detection. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 19830–19843, 2023. 2

    work page 2023

  5. [5]

    Diffusion policy: Visuomotor policy learning via action dif- fusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action dif- fusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 2, 6, 8

    work page 2023

  6. [6]

    Bayesian imitation learning for end-to-end mo- bile manipulation

    Yuqing Du, Daniel Ho, Alex Alemi, Eric Jang, and Mohi Khansari. Bayesian imitation learning for end-to-end mo- bile manipulation. In International Conference on Machine Learning, pages 5531–5546. PMLR, 2022. 1

    work page 2022

  7. [7]

    Implicit behavioral cloning

    Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on robot learning , pages 158–168. PMLR, 2022. 2, 5, 6

    work page 2022

  8. [8]

    Recurrent world models facilitate policy evolution

    David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018. 3

    work page 2018

Show all 50 references
  1. [9]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. arXiv preprint arXiv:1912.01603, 2019

    1912 arXiv

  2. [10]

    Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    2010 arXiv

  3. [11]

    Deep hierarchical planning from pixels

    Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. Advances in Neural Information Processing Systems, 35:26091–26104,

  4. [12]

    Teach a robot to fish: Versatile imitation from one minute of demonstrations

    Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023. 1

    2023 arXiv

  5. [13]

    On pre-training for visuo-motor control: Re- visiting a learning-from-scratch baseline

    Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiao- long Wang. On pre-training for visuo-motor control: Re- visiting a learning-from-scratch baseline. arXiv preprint arXiv:2212.05749, 2022. 1

    2022 arXiv

  6. [14]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2

    2022

  7. [15]

    Model-based imitation learning for urban driving

    Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. Advances in Neural Information Process- ing Systems, 35:20703–20716, 2022. 3

    2022

  8. [16]

    Gaia-1: A generative world model for au- tonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving. arXiv preprint arXiv:2309.17080 , 2023. 3

    2023 arXiv

  9. [17]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and An- drew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Let- ters, 5(2):3019–3026, 2020. 5, 4

    2020

  10. [18]

    Coarse-to-fine imitation learning: Robot ma- nipulation from a single demonstration

    Edward Johns. Coarse-to-fine imitation learning: Robot ma- nipulation from a single demonstration. In 2021 IEEE in- ternational conference on robotics and automation (ICRA) , pages 4613–4619. IEEE, 2021. 1

    2021

  11. [19]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  12. [20]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Fos- ter, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kol- lar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Ope...

    2024 arXiv

  13. [21]

    Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy

    Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, and Tao Kong. Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy. IEEE Robotics and Automation Letters , 2025

    2025

  14. [22]

    Rdt-1b: a diffusion foundation model for bimanual manipu- lation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipu- lation. arXiv preprint arXiv:2410.07864, 2024. 2

    2024 arXiv

  15. [23]

    Manigaussian: Dynamic gaus- sian splatting for multi-task robotic manipulation

    Guanxing Lu, Shiyi Zhang, Ziwei Wang, Changliu Liu, Ji- wen Lu, and Yansong Tang. Manigaussian: Dynamic gaus- sian splatting for multi-task robotic manipulation. In Eu- ropean Conference on Computer Vision , pages 349–366. Springer, 2024. 6

    2024

  16. [24]

    What mat- ters in learning from offline human demonstrations for robot manipulation

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiri- any, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart´ın-Mart´ın. What mat- ters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 202...

    2021 arXiv

  17. [25]

    Diffusion-dice: In-sample diffusion guid- ance for offline reinforcement learning

    Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In-sample diffusion guid- ance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024. 2

    2024 arXiv

  18. [26]

    Learning agile robotic locomotion skills by imitating animals

    Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784, 2020. 1

    2004 arXiv

  19. [27]

    Dexmv: Imitation learning for dexterous manipulation from human videos

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Rui- han Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision , pages 570–587. Springer, 2022. 1, 2

    2022

  20. [28]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giu- lia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017. 5

    2017 arXiv

  21. [29]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 5

    2017 arXiv

  22. [30]

    Masked world models for visual control

    Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332–1344. PMLR, 2023. 3

    2023

  23. [31]

    Behavior transformers: Cloning k modes with one stone

    Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Al- tanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022. 1, 2

    2022

  24. [32]

    On bringing robots home

    Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint arXiv:2311.16098, 2023. 1

    2023 arXiv

  25. [33]

    Perceiver- actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver- actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning , pages 785–799. PMLR,

  26. [34]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. arXiv:2010.02502, 2020. 6

    2010 arXiv

  27. [35]

    Vrl3: A data-driven framework for visual deep reinforce- ment learning

    Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vrl3: A data-driven framework for visual deep reinforce- ment learning. Advances in Neural Information Processing Systems, 35:32974–32988, 2022. 5

    2022

  28. [36]

    Mim- icplay: Long-horizon imitation learning by watching human play

    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei- Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mim- icplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023. 1, 2

    2023 arXiv

  29. [37]

    Drivedreamer: Towards real-world- drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jia- gang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world- drive world models for autonomous driving. In European Conference on Computer Vision , pages 55–72. Springer,

  30. [38]

    Dif- fusion policies as an expressive policy class for offline rein- forcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Dif- fusion policies as an expressive policy class for offline rein- forcement learning. arXiv preprint arXiv:2208.06193, 2022. 2, 3

    2022 arXiv

  31. [39]

    4d gaussian splatting for real-time dynamic scene rendering

    Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 20310–2032...

    2024

  32. [40]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning , pages 2226–2240. PMLR, 2023. 3

    2023

  33. [41]

    Policy representation via diffusion prob- ability model for reinforcement learning

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion prob- ability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023. 2

    2023 arXiv

  34. [42]

    Diffusion models: A comprehensive survey of methods and applications

    Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Run- sheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4): 1–39, 2023. 2

    2023

  35. [43]

    General- ized animal imitator: Agile locomotion with versatile motion prior

    Ruihan Yang, Zhuoqun Chen, Jianhan Ma, Chongyi Zheng, Yiyu Chen, Quan Nguyen, and Xiaolong Wang. General- ized animal imitator: Agile locomotion with versatile motion prior. arXiv preprint arXiv:2310.01408, 2023. 1

    2023 arXiv

  36. [44]

    Gnfactor: Multi-task real robot learning with generalizable neural feature fields

    Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on Robot Learning, pages 284–301. PMLR, 2023. 6

    2023

  37. [45]

    3d diffusion policy: Gen- eralizable visuomotor policy learning via simple 3d repre- sentations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Gen- eralizable visuomotor policy learning via simple 3d repre- sentations. In Proceedings of Robotics: Science and Systems (RSS), 2024. 2, 5, 6, 8, 3

    2024

  38. [46]

    Point cloud matters: Rethinking the impact of different observation spaces on robot learn- ing

    Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learn- ing. Advances in Neural Information Processing Systems , 37:77799–77830, 2025. 2 Spatial-Temporal Aware Visuomotor...

    2025

  39. [47]

    Stacking Cups

    Video The attached video demonstrates the application of our 4D Diffusion Policy, which is capable of handling com- plex tasks in real-world through high-level spatial-temporal awareness. We present an expert demonstration and evalu- ate the performance of DP4 after learning f...

  40. [48]

    pointnet

    Additional Implementation Details The proposed 4D Diffusion Policy (DP4) comprises three key components: Multi-level 3D Spatial Awareness, 4D Spatiotemporal Awareness, and Diffusion-based Decision. Below, we provide a detailed description of the implemen- tation of each compon...

  41. [49]

    Grasping Bottles

    Details of Simulation Tasks For the simulation experiments, we selected tasks from var- ious domains, covering a broad spectrum of robotic skills. These tasks include both challenging scenarios, such as bi- manual manipulation, deformable object manipulation, and articulated o...

  42. [50]

    grasping bottles

    Details of Real-Robot Tasks In the experiments, we conduct three primary tasks, along with three additional ones that include distracting objects. The “grasping bottles” task requires the agent to pick up a bottle from a table, a challenge due to the precise coordina- tion and...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.