pith. sign in

arxiv: 2303.04137 · v5 · submitted 2023-03-07 · 💻 cs.RO

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

Cheng Chi , Zhenjia Xu , Siyuan Feng , Eric Cousineau , Yilun Du , Benjamin Burchfiel , Russ Tedrake , Shuran Song This is my paper

Pith reviewed 2026-05-13 00:15 UTC · model grok-4.3

classification 💻 cs.RO
keywords diffusion policyvisuomotor policy learningaction diffusionrobot manipulationdenoising diffusion processmultimodal action distributionspolicy learning
0
0 comments X

The pith

Robot visuomotor policies can be represented as conditional denoising diffusion processes to outperform state-of-the-art methods by 46.9% on average.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that modeling a robot's policy for visuomotor control as a conditional denoising diffusion process produces more effective behavior generation than existing approaches. A sympathetic reader would care because it provides concrete advantages in dealing with the inherent uncertainty and complexity of robot actions from visual inputs. The method learns the score function gradient of the action distribution and refines actions through iterative stochastic steps, leading to better results on manipulation benchmarks. This could open doors to policies that are more robust and easier to train for physical robots.

Core claim

Diffusion Policy learns the gradient of the action-distribution score function and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevin dynamics steps. The diffusion formulation yields powerful advantages when used for robot policies, including gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting impressive training stability. To fully unlock the potential, the paper incorporates receding horizon control, visual conditioning, and the time-series diffusion transformer, resulting in consistent outperformance over existing state-of-the-art robot learning methods with an 46

What carries the argument

The conditional denoising diffusion process that represents the visuomotor policy and generates actions by starting from noise and iteratively denoising guided by visual observations.

If this is right

  • Gracefully handles multimodal action distributions
  • Suitable for high-dimensional action spaces
  • Exhibits impressive training stability
  • Supports receding horizon control for improved performance

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Diffusion-based policies may scale to more complex tasks or different modalities like language-conditioned actions.
  • The stability during training suggests diffusion models could replace less stable generative methods in other control applications.
  • Further work on accelerating the Langevin dynamics steps could broaden the applicability to faster control loops.

Load-bearing premise

The iterative stochastic Langevin dynamics steps required for inference can be executed at a rate compatible with real-time closed-loop control on physical robot hardware without unacceptable latency or instability.

What would settle it

Observing the actual inference latency and closed-loop stability when deploying the diffusion policy on physical robot hardware for the benchmark tasks.

read the original abstract

This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process. We benchmark Diffusion Policy across 12 different tasks from 4 different robot manipulation benchmarks and find that it consistently outperforms existing state-of-the-art robot learning methods with an average improvement of 46.9%. Diffusion Policy learns the gradient of the action-distribution score function and iteratively optimizes with respect to this gradient field during inference via a series of stochastic Langevin dynamics steps. We find that the diffusion formulation yields powerful advantages when used for robot policies, including gracefully handling multimodal action distributions, being suitable for high-dimensional action spaces, and exhibiting impressive training stability. To fully unlock the potential of diffusion models for visuomotor policy learning on physical robots, this paper presents a set of key technical contributions including the incorporation of receding horizon control, visual conditioning, and the time-series diffusion transformer. We hope this work will help motivate a new generation of policy learning techniques that are able to leverage the powerful generative modeling capabilities of diffusion models. Code, data, and training details is publicly available diffusion-policy.cs.columbia.edu

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Diffusion Policy, representing visuomotor robot policies as conditional denoising diffusion processes. It reports benchmarking results on 12 tasks from 4 robot manipulation benchmarks, with consistent outperformance of prior state-of-the-art methods by an average of 46.9%. The method adapts diffusion-model inference via stochastic Langevin dynamics, with technical extensions for receding-horizon control, visual conditioning, and a time-series diffusion transformer. Code, data, and training details are released publicly.

Significance. If the empirical results hold under rigorous re-evaluation, the work demonstrates that diffusion-based generative modeling can yield substantial gains in robot policy learning, especially for multimodal action distributions and high-dimensional spaces, while offering training stability advantages. The public release of code and data is a notable strength that supports reproducibility and extension by the community.

major comments (2)
  1. [§4 (Experiments) and Table 1] §4 (Experiments) and Table 1: The central claim of a 46.9% average improvement across 12 tasks provides no statistical significance tests, standard deviations across random seeds, or explicit confirmation that all baselines were re-implemented with equivalent hyperparameter search and evaluation protocols; this weakens confidence that the reported gains are robust rather than sensitive to implementation details.
  2. [§3.3 (Inference Procedure)] §3.3 (Inference Procedure): The assertion that the iterative denoising steps are compatible with real-time closed-loop control on physical hardware is load-bearing for the practical contribution, yet no wall-clock latency measurements, control-frequency benchmarks, or hardware-specific timing results are reported to substantiate this.
minor comments (2)
  1. [Abstract] Abstract: The specific names of the 4 benchmarks and 12 tasks are not listed, which would allow readers to immediately assess task diversity and difficulty.
  2. [§3.1] §3.1: The notation for the conditional score function and the precise form of the visual conditioning could be made more explicit with an additional equation or diagram for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their positive assessment of the work and recommendation for minor revision. We address each major comment point by point below.

read point-by-point responses

  1. Referee: [§4 (Experiments) and Table 1] The central claim of a 46.9% average improvement across 12 tasks provides no statistical significance tests, standard deviations across random seeds, or explicit confirmation that all baselines were re-implemented with equivalent hyperparameter search and evaluation protocols; this weakens confidence that the reported gains are robust rather than sensitive to implementation details.

    Authors: We appreciate the referee's emphasis on rigorous empirical validation. Our baseline re-implementations followed the original papers' protocols with hyperparameter tuning, and the public code release enables independent verification of these details. We agree, however, that explicitly reporting standard deviations across random seeds and statistical significance tests would strengthen confidence in the results. In the revised manuscript we will update Table 1 and §4 to include mean performance with standard deviations over multiple seeds and paired statistical tests for the key comparisons. revision: yes

  2. Referee: [§3.3 (Inference Procedure)] The assertion that the iterative denoising steps are compatible with real-time closed-loop control on physical hardware is load-bearing for the practical contribution, yet no wall-clock latency measurements, control-frequency benchmarks, or hardware-specific timing results are reported to substantiate this.

    Authors: We agree that concrete timing measurements are necessary to fully substantiate the claim of real-time compatibility. The inference procedure was designed with a fixed number of denoising steps and receding-horizon control precisely to enable closed-loop operation. In the revised manuscript we will add wall-clock latency results, achieved control frequencies, and hardware specifications from our experimental platforms to §3.3 and the experimental section. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper adapts established conditional diffusion models to represent visuomotor policies as denoising processes. All load-bearing claims are empirical benchmark results (46.9% average improvement across 12 tasks) rather than derivations that reduce by construction to fitted parameters, self-citations, or renamed inputs. The receding-horizon, visual conditioning, and transformer components are presented as standard extensions of the diffusion framework without internal self-definition or fitted-input-as-prediction patterns. No equations or steps equate outputs to inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The approach rests on standard assumptions of diffusion models and imitation learning from demonstrations; no new physical entities are postulated.

free parameters (2)
  • number of diffusion steps
    Hyperparameter controlling the number of denoising iterations during training and inference.
  • noise schedule parameters
    Parameters defining how noise is added and removed, chosen as part of model design.
axioms (2)
  • domain assumption Robot action distributions can be effectively modeled as the reverse of a forward diffusion process conditioned on visual observations.
    Invoked in the definition of the policy as a conditional denoising diffusion process.
  • domain assumption Demonstration data provides sufficient coverage for supervised training of the score function.
    Implicit in the imitation-learning setup used to train the diffusion policy.

pith-pipeline@v0.9.0 · 5519 in / 1377 out tokens · 44278 ms · 2026-05-13T00:15:44.928352+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 57 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution

    cs.CL 2023-09 unverdicted novelty 8.0

    Promptbreeder evolves both task prompts and the mutation prompts that improve them using LLMs, outperforming Chain-of-Thought and Plan-and-Solve on arithmetic and commonsense reasoning benchmarks.

  2. Point Tracking Improves World Action Models

    cs.RO 2026-05 unverdicted novelty 7.0

    JOPAT jointly models pixels, point tracks, and actions in a diffusion transformer and reports gains over pixel-only baselines on long-horizon robot tasks with occlusion and off-screen motion.

  3. SkiP: When to Skip and When to Refine for Efficient Robot Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    SkiP introduces action relabeling and Motion Spectrum Keying to skip redundant steps in robot trajectories, cutting executed steps by 15-40% while maintaining success rates across 72 simulated and 3 real tasks.

  4. DSSP: Diffusion State Space Policy with Full-History Encoding

    cs.RO 2026-05 conditional novelty 7.0

    DSSP is a history-conditioned diffusion state space policy that uses SSMs to encode full observation streams with an auxiliary dynamics objective and hierarchical fusion, achieving SOTA results with reduced model size...

  5. Dynamic Execution Commitment of Vision-Language-Action Models

    cs.CV 2026-05 unverdicted novelty 7.0

    A3 determines the execution horizon in VLA models as the longest prefix of actions that passes consensus-based verification and sequential consistency checks.

  6. OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation

    cs.RO 2026-05 unverdicted novelty 7.0

    OA-WAM uses persistent address vectors and dynamic content vectors in object slots to enable addressable world-action prediction, improving robustness on manipulation benchmarks under scene changes.

  7. Latent State Design for World Models under Sufficiency Constraints

    cs.AI 2026-05 unverdicted novelty 7.0

    World models succeed when their latent states are built to meet task-specific sufficiency constraints rather than preserving the maximum amount of information.

  8. Action Agent: Agentic Video Generation Meets Flow-Constrained Diffusion

    cs.RO 2026-05 unverdicted novelty 7.0

    Action Agent pairs LLM-driven video generation with a flow-constrained diffusion transformer to produce velocity commands, raising video success to 86% and delivering 64.7% real-world navigation on a Unitree G1 humanoid.

  9. Atomic-Probe Governance for Skill Updates in Compositional Robot Policies

    cs.RO 2026-04 unverdicted novelty 7.0

    A cross-version swap protocol reveals dominant skills that swing composition success by up to 50 percentage points, and an atomic probe with selective revalidation governs updates at lower cost than always re-testing ...

  10. CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policies

    cs.CV 2026-04 unverdicted novelty 7.0

    CF-VLA uses a coarse initialization over endpoint velocity followed by single-step refinement to achieve strong performance with low inference steps on CALVIN, LIBERO, and real-robot tasks.

  11. Characterizing Vision-Language-Action Models across XPUs: Constraints and Acceleration for On-Robot Deployment

    cs.RO 2026-04 unverdicted novelty 7.0

    VLA models exhibit a compute-bound VLM phase followed by a memory-bound action phase on edge hardware; DP-Cache and V-AEFusion reduce redundancy and enable pipeline parallelism for up to 6x speedup on NPUs with margin...

  12. Mask World Model: Predicting What Matters for Robust Robot Policy Learning

    cs.RO 2026-04 unverdicted novelty 7.0

    Mask World Model predicts semantic mask dynamics with video diffusion and integrates it with a diffusion policy head, outperforming RGB world models on LIBERO and RLBench while showing better real-world generalization...

  13. Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation

    cs.RO 2026-04 unverdicted novelty 7.0

    ReV is a referring-aware visuomotor policy using coupled diffusion heads for real-time trajectory replanning in robotic manipulation, trained solely via targeted perturbations to expert demonstrations and achieving hi...

  14. Receding-Horizon Control via Drifting Models

    cs.AI 2026-04 unverdicted novelty 7.0

    Drifting MPC produces a unique distribution over trajectories that trades off data support against optimality and enables efficient receding-horizon planning under unknown dynamics.

  15. UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

    cs.RO 2026-02 unverdicted novelty 7.0

    UniLACT improves VLA models by adding depth-aware unified latent action pretraining that outperforms RGB-only baselines on seen and unseen manipulation tasks.

  16. Information Filtering via Variational Regularization for Robot Manipulation

    cs.RO 2026-01 unverdicted novelty 7.0

    Variational Regularization imposes an adaptive information bottleneck on noisy intermediate features in DP3-UNet and DP3-DiT policies, consistently raising task success rates on RoboTwin2.0, Adroit, and MetaWorld whil...

  17. Multimodal Diffusion Forcing for Forceful Manipulation

    cs.RO 2025-11 unverdicted novelty 7.0

    Multimodal Diffusion Forcing trains a diffusion model on partially masked multimodal robot trajectories to learn temporal and cross-modal dependencies for forceful manipulation.

  18. One Step Diffusion via Shortcut Models

    cs.LG 2024-10 conditional novelty 7.0

    Shortcut models enable high-quality single or few-step sampling in diffusion models with one network and training phase by conditioning on desired step size.

  19. RoboDreamer: Learning Compositional World Models for Robot Imagination

    cs.RO 2024-04 unverdicted novelty 7.0

    RoboDreamer factorizes video generation using language primitives to achieve compositional generalization in robot world models, outperforming monolithic baselines on unseen goals in RT-X.

  20. Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

    cs.RO 2023-10 conditional novelty 7.0

    SuSIE uses a finetuned InstructPix2Pix diffusion model to propose subgoal images that guide a low-level goal-conditioned policy, achieving SOTA zero-shot performance on CALVIN and real-world manipulation.

  21. Mechanisms of Misgeneralization in Physical Sequence Modeling

    cs.LG 2026-05 unverdicted novelty 6.0

    Generative sequence models for physical tasks exhibit physical misgeneralization where local prediction errors propagate through physical measurements to distort aggregate distributions over quantities like distance o...

  22. COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphones

    cs.RO 2026-05 unverdicted novelty 6.0

    COBALT provides scalable cloud infrastructure for crowdsourced robot teleoperation via smartphones, supporting concurrent users with low latency and enabling collection of a 7500+ demonstration dataset validated throu...

  23. COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphones

    cs.RO 2026-05 conditional novelty 6.0

    COBALT enables scalable crowdsourced teleoperation of robots using smartphones, supporting concurrent users with low latency and yielding a 7500+ demonstration dataset validated on imitation learning tasks.

  24. Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

    cs.LG 2026-05 unverdicted novelty 6.0

    Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.

  25. Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation

    cs.CV 2026-05 conditional novelty 6.0

    VLA-AD distills 7B VLA teachers into 158M students using offline VLM semantic guidance on task phases and directions, matching teacher performance on LIBERO with 44x size reduction and 3.28x speedup.

  26. Learning Sim-Grounded Policies for Bimanual Rope Manipulation from Human Teleoperation Data

    cs.RO 2026-05 conditional novelty 6.0

    A simulation-grounded state policy using 3D particle dynamics outperforms an egocentric vision policy by 30.8% in L1 error on unseen rope configurations for bimanual manipulation from limited human data.

  27. Enforcing Constraints in Generative Sampling via Adaptive Correction Scheduling

    cs.LG 2026-05 unverdicted novelty 6.0

    Adaptive correction scheduling for hard constraints in generative sampling recovers 71% of stepwise projection benefits using 75% fewer corrections by focusing on trajectory-perturbing steps.

  28. OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

    cs.LG 2026-05 unverdicted novelty 6.0

    OGPO is a sample-efficient off-policy method for full finetuning of generative control policies that reaches SOTA on robotic manipulation tasks and can recover from poor behavior-cloning initializations without expert data.

  29. Atomic-Probe Governance for Skill Updates in Compositional Robot Policies

    cs.RO 2026-04 unverdicted novelty 6.0

    Empirical study on robosuite tasks reveals a dominant-skill effect in compositions and shows that an atomic probe approximates full revalidation for skill updates at much lower cost.

  30. AsyncShield: A Plug-and-Play Edge Adapter for Asynchronous Cloud-based VLA Navigation

    cs.RO 2026-04 unverdicted novelty 6.0

    AsyncShield restores VLA geometric intent from latency via kinematic pose mapping and uses PPO-Lagrangian to balance tracking with LiDAR safety constraints in a plug-and-play module.

  31. dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model

    cs.RO 2026-04 unverdicted novelty 6.0

    A discrete diffusion model tokenizes multimodal robotic data and uses a progress token to predict future states and task completion for scalable policy evaluation.

  32. Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

    cs.RO 2026-04 unverdicted novelty 6.0

    Hi-WM uses human interventions inside an action-conditioned world model with rollback and branching to generate dense corrective data, raising real-world success by 37.9 points on average across three manipulation tasks.

  33. FASTER: Value-Guided Sampling for Fast RL

    cs.LG 2026-04 unverdicted novelty 6.0

    FASTER models multi-candidate denoising as an MDP and trains a value function to filter actions early, delivering the performance of full sampling at lower cost in diffusion RL policies.

  34. Accelerating trajectory optimization with Sobolev-trained diffusion policies

    cs.LG 2026-04 unverdicted novelty 6.0

    Sobolev-trained diffusion policies using trajectories and feedback gains provide warm-starts that reduce trajectory optimization solving time by 2x to 20x while avoiding compounding errors.

  35. SpaceDex: Generalizable Dexterous Grasping in Tiered Workspaces

    cs.RO 2026-04 unverdicted novelty 6.0

    SpaceDex achieves 63% success grasping unseen objects in tiered workspaces via VLM spatial planning and arm-hand feature separation, beating a 39% tabletop baseline in 100 real trials.

  36. Positive-Only Drifting Policy Optimization

    cs.LG 2026-04 unverdicted novelty 6.0

    PODPO is a likelihood-free generative policy optimization method for online RL that steers actions to high-return regions using only positive-advantage samples and local contrastive drifting.

  37. AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence

    cs.RO 2026-04 unverdicted novelty 6.0

    AffordGen generates affordance-aware manipulation demonstrations from 3D mesh correspondences to train policies with zero-shot generalization to novel objects.

  38. InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation

    cs.RO 2026-02 unverdicted novelty 6.0

    InCoM achieves 23-28% higher success rates in mobile manipulation tasks by inferring motion intent for adaptive perception and decoupling base-arm action generation.

  39. mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

    cs.RO 2025-12 unverdicted novelty 6.0

    mimic-video combines internet video pretraining with a flow-matching decoder to achieve state-of-the-art robotic manipulation performance with 10x better sample efficiency than vision-language-action models.

  40. X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations

    cs.RO 2025-11 unverdicted novelty 6.0

    X-Diffusion adapts Ambient Diffusion to selectively train on noised human actions for cross-embodiment robot policies, yielding 16% higher average success rates than naive co-training or manual filtering across five r...

  41. DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control

    cs.RO 2025-02 unverdicted novelty 6.0

    DexVLA combines a scaled diffusion action expert with embodiment curriculum learning to achieve better generalization and performance than prior VLA models on diverse robot hardware and long-horizon tasks.

  42. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

    cs.CV 2024-12 unverdicted novelty 6.0

    Video Prediction Policy conditions robot action learning on future-frame predictions inside fine-tuned video diffusion models, yielding 18.6% relative gains on Calvin ABC-D and 31.6% higher real-world success rates.

  43. RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields

    cs.RO 2024-12 unverdicted novelty 6.0

    A deep RL vulnerability-prediction policy trained in semantic embedding space finds up to 23% more unique robot manipulation failures than vision-language baselines and enables more efficient fine-tuning.

  44. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

    cs.RO 2024-11 unverdicted novelty 6.0

    DINO-WM builds world models on pre-trained DINOv2 features to enable zero-shot planning from offline data without rewards or demonstrations.

  45. Language Conditioned Multi-Finger Dexterous Manipulation Enabled by Physical Compliance and Switching of Controllers

    cs.RO 2024-10 unverdicted novelty 6.0

    A hybrid event-driven switching system pairs VLA models with lightweight dexterous policies on a compliant anthropomorphic hand to perform language-conditioned multi-finger tasks with cross-embodiment modularity.

  46. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots

    cs.RO 2024-06 unverdicted novelty 6.0

    RoboCasa supplies a large-scale kitchen simulator, generative assets, 100 tasks, and automated data pipelines that produce a clear scaling trend in imitation learning for generalist robots.

  47. 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

    cs.RO 2024-02 conditional novelty 6.0

    3D Diffuser Actor unifies diffusion policies with 3D scene features to set new state-of-the-art results on RLBench and CALVIN robot benchmarks.

  48. Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

    cs.RO 2023-12 conditional novelty 6.0

    A GPT-style model pre-trained on large video datasets achieves 94.9% success on CALVIN multi-task manipulation and 85.4% zero-shot generalization, outperforming prior baselines.

  49. Training Diffusion Models with Reinforcement Learning

    cs.LG 2023-05 unverdicted novelty 6.0

    DDPO uses policy gradients on the denoising process to optimize diffusion models for arbitrary rewards like human feedback or compressibility.

  50. Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

    cs.RO 2026-05 unverdicted novelty 5.0

    PACTS jointly model action trajectories and predicate belief trajectories in a single generative policy, enabling zero-shot skill composition via symbolic planning without retraining.

  51. Dynamic Execution Commitment of Vision-Language-Action Models

    cs.CV 2026-05 unverdicted novelty 5.0

    A3 adaptively selects verifiable action prefixes in VLA models using group-sampled consensus and conditional re-decoding to balance robustness and speed without manual horizon tuning.

  52. World-Value-Action Model: Implicit Planning for Vision-Language-Action Systems

    cs.RO 2026-04 unverdicted novelty 5.0

    The World-Value-Action model enables implicit planning for VLA systems by performing inference over a learned latent representation of high-value future trajectories instead of direct action prediction.

  53. OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction

    cs.RO 2026-04 unverdicted novelty 5.0

    OmniUMI introduces a multimodal handheld interface that synchronously records RGB, depth, trajectory, tactile, internal grasp force, and external wrench data for training diffusion policies on contact-rich robot manipulation.

  54. D2 Actor Critic: Diffusion Actor Meets Distributional Critic

    cs.LG 2025-10 unverdicted novelty 5.0

    D2AC combines a diffusion actor with a distributional critic via fused distributional RL and clipped double Q-learning to reach state-of-the-art results on 18 hard control benchmarks including Humanoid, Dog, and Shadow Hand.

  55. A Survey on Vision-Language-Action Models: An Action Tokenization Perspective

    cs.RO 2025-07 unverdicted novelty 5.0

    The survey frames VLA models as pipelines that generate progressively grounded action tokens and classifies those tokens into eight types to guide future development.

  56. NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks

    cs.RO 2025-04 unverdicted novelty 5.0

    NORA is a compact 3B-parameter VLA model trained on 970k robot demonstrations that outperforms larger VLA models in embodied tasks while using significantly less computational resources.

  57. MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

    cs.RO 2023-10 unverdicted novelty 5.0

    MimicGen creates over 50K robot demonstrations from roughly 200 human ones, allowing imitation learning to achieve strong performance on complex long-horizon tasks like assembly and coffee preparation.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 54 Pith papers · 6 internal anchors

  1. [1]

    Is Conditional Generative Modeling all you need for Decision-Making?

    Ajay A, Du Y , Gupta A, Tenenbaum J, Jaakkola T and Agrawal P (2022) Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 . Argall BD, Chernova S, Veloso M and Browning B (2009) A survey of robot learning from demonstration. Robotics and autonomous systems 57(5): 469–483. 14 Atkeson CG and Schaal S (1997) Ro...

    work page internal anchor Pith review arXiv 2022

  2. [2]

    pp. 12–20. Avigal Y , Berscheid L, Asfour T, Kr¨oger T and Goldberg K (2022) Speedfolding: Learning efficient bimanual folding of garments. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 1–8. Bishop CM (1994) Mixture density networks. Aston University. Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp ...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Deng J, Dong W, Socher R, Li LJ, Li K and Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition . Ieee, pp. 248–255. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al. (2020) An image is worth 16x16...

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  4. [4]

    Elucidating the Design Space of Diffusion-Based Generative Models

    Karras T, Aittala M, Aila T and Laine S (2022) Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 . Khatib O (1987) A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation 3(1): 43–53. DOI:10. 1109/JRA.1987.1087068. Con...

    work page internal anchor Pith review arXiv 2022

  5. [5]

    (2021) Learning transferable visual models from natural language supervision

    Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et al. (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning . PMLR, pp. 8748–8763. Rahmatizadeh R, Abolghasemi P, B ¨ol¨oni L and Levine S (2018) Vision-based multi-task manipulation for...

    work page 2021

  6. [6]

    In: Proceedings of Robotics: Science and Systems (RSS)

    Reuss M, Li M, Jia X and Lioutikov R (2023) Goal-conditioned imitation learning using score-based diffusion policies. In: Proceedings of Robotics: Science and Systems (RSS). Ridnik T, Ben-Baruch E, Noy A and Zelnik-Manor L (2021) Imagenet-21k pretraining for the masses. Ronneberger O, Fischer P and Brox T (2015) U-net: Convolu- tional networks for biomedi...

    work page 2023

  7. [7]

    Consistency Models

    Springer. Sohl-Dickstein J, Weiss E, Maheswaranathan N and Ganguli S (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning. Song J, Meng C and Ermon S (2021) Denoising diffusion implicit models. In: International Conference on Learning Representations. Song Y , Dhariwal P, Chen M and Sutske...

    work page internal anchor Pith review arXiv 2015

  8. [8]

    In: 2019 IEEE 58th Conference on Decision and Control (CDC)

    Subramanian J and Mahajan A (2019) Approximate information state for partially observed systems. In: 2019 IEEE 58th Conference on Decision and Control (CDC). IEEE, pp. 1629–

    work page 2019

  9. [9]

    Conditional energy- based models for implicit policies: The gap between theory and prac- tice,

    Ta DN, Cousineau E, Zhao H and Feng S (2022) Conditional energy-based models for implicit policies: The gap between theory and practice. arXiv preprint arXiv:2207.05824 . Tancik M, Srinivasan P, Mildenhall B, Fridovich-Keil S, Raghavan N, Singhal U, Ramamoorthi R, Barron J and Ng R (2020) Fourier features let networks learn high frequency functions in low...

    work page arXiv 2022

  10. [10]

    Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

    Wang Z, Hunt JJ and Zhou M (2022) Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 . Wang Z, Hunt JJ and Zhou M (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. In: The Eleventh International Conference on Learning Representations. URL https://op...

    work page internal anchor Pith review arXiv 2022

  11. [11]

    Diffusion Policy outperforms LSTM-GMM Mandlekar et al

    Data Efficiency Ablation Study. Diffusion Policy outperforms LSTM-GMM Mandlekar et al. (2021) at every training dataset size. except Push-T. Performance reported for DiffusionPolicy-C on Push-T in Tab. 1 used impaiting instead of FiLM. On simulation benchmarks, we used the iDDPM algorithm Nichol and Dhariwal (2021) with the same 100 denoising diffusion it...

    work page 2021

  12. [12]

    Hyperparameters for CNN-based Diffusion Policy Ctrl: position or velocity control To: observation horizon Ta: action horizon Tp: action prediction horizon ImgRes: environment observation resolution (Camera views x W x H) CropRes: random crop resolution #D-Params: diffusion network number of parameters in millions #V-Params: vision encoder number of parame...

    work page 2021

  13. [13]

    (2021)) B.2 Performance Improvement Calculation For each task i (column) reported in Tab

    Hyperparameters for Transformer-based Diffusion Policy Ctrl: position or velocity control To: observation horizon Ta: action horizon Tp: action prediction horizon #D-Params: diffusion network number of parameters in millions #V-Params: vision encoder number of parameters in millions Emb Dim: transformer token embedding dimension Attn Drp: transformer atte...

    work page 2021

  14. [14]

    To ensure the consistency of initial conditions, we carefully adjusted the pose of the T block and the robot according to overlayed images from the top-down camera

    Each method is evaluated for 20 episodes, all starting from the same set of initial conditions. To ensure the consistency of initial conditions, we carefully adjusted the pose of the T block and the robot according to overlayed images from the top-down camera. Each evaluation episode is terminated by either keeping the end-effector within the end-zone for...

    work page 1987