Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AC-DiT shows that conditioning whole-body actions on a latent mobility feature and reweighting 2D/3D inputs by language similarity improves end-to-end mobile manipulation, beating baselines in simulation and on a real robot.

desk verdict Two sensible mechanisms for mobile manipulation, but the evidence as reported doesn't yet prove the second one works. read the letter →

arxiv 2507.01961 v3 pith:OWOIWDFA submitted 2025-07-02 cs.RO cs.AI

Sixiang Chen , Jiaming Liu , Siyuan Qian , Han Jiang , Lily Li , Renrui Zhang , Zhuoyang Liu , Chenyang Gu
show 4 more authors
Chengkai Hou Pengwei Wang Zhongyuan Wang Shanghang Zhang
This is my paper · ORCID
classification cs.ROcs.AI
keywords mobilemanipulationdiffusiontransformerwhole-bodycoordinationmultimodalperception2D/3Dfusionimitationlearningvision-language-actionmodelbimanual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that end-to-end mobile manipulation fails for two identifiable reasons: the arm is planned without knowing what the base is about to do, and the policy uses the same visual modality mix regardless of task stage. AC-DiT addresses both by pretraining a lightweight diffusion-transformer head to predict base motion alone, then feeding the resulting latent mobility feature into the cross-attention of the main DiT head that predicts base and arm actions together. It also computes cosine similarity between language features and each projected 2D view and 3D point cloud, turns those similarities into per-stage weights, and re-weights the visual tokens before action prediction. The reported results are higher success rates than imitation-learning and VLA baselines on ManiSkill-HAB, RoboTwin, and four real-world long-horizon tasks. If true, this gives mobile manipulation a concrete way to coordinate locomotion and manipulation and to choose the right perception for each phase of a task.

What carries the argument

The central mechanism is the mobility-to-body conditioning loop inside a diffusion transformer (DiT), a transformer that iteratively denoises action sequences. A lightweight DiT head is pretrained to predict only base actions, and its final-block tokens across five denoising steps become a latent mobility feature $F_m$; this feature is concatenated with the multimodal visual and language features and injected through cross-attention into the main body-action DiT head. The second mechanism is perception-aware multimodal adaptation, which projects each 2D view and the 3D point cloud into a shared space with the language feature and uses normalized cosine similarities as per-stage importance weights applied to the visual tokens. The pair carries the argument: one injects base-motion priors into whole-body action prediction, and the other selects stage-appropriate perception.

What would settle it

Train AC-DiT exactly as in the paper but replace the perception weights with uniform constants while keeping every projection MLP and the conditioning injection unchanged; if the mean success rate on ManiSkill-HAB stays near the reported 49.0 percent rather than falling, the similarity-based weights are not causally responsible for the gains.

Watch

Extended reading notes

Core claim

The authors aim to establish that whole-body coordination in mobile manipulation can be improved by explicitly conditioning arm actions on base-motion priors, and that stage-appropriate perception can be learned by weighting 2D and 3D inputs against the language instruction. In their architecture, a lightweight mobility DiT head (about 170 million parameters) is pretrained on only the base's linear and angular velocities; the tokens from its final DiT block over the five denoising steps are concatenated into a latent mobility feature $F_m$, which is combined with multimodal features and injected into the cross-attention of the main mobile-manipulation DiT head. A second module projects the three camera views and the point cloud into a shared space with the language feature and computes weights $w_i^f, w_i^l, w_i^r, w_i^p$ as cosine similarities, producing reweighted visual features that vary by stage. The paper reports mean success rates of 55.6 percent on ManiSkill-HAB across seven tasks and 90.1 percent on RoboTwin across six dual-arm tasks, above its strongest baselines, and higher subtask success than ACT and π0 on all four real-world tasks.

Load-bearing premise

The load-bearing premise is that the cosine similarity between a projected camera view or point cloud and the language instruction measures how much that modality should matter at the current stage; the paper's ablations never test whether the learned weights themselves, rather than the added projection layers or the conditioning architecture, cause the reported improvements.

Editorial extensions

If this is right

  • Mobile manipulation policies can be trained end-to-end without a separate planner and still anticipate the base's effect on the arm, reducing error accumulation in high-degree-of-freedom control.
  • A policy can switch its perceptual emphasis during a single episode, using 2D semantics for locating objects and 3D geometry for contacting them, without being told explicitly which stage it is in.
  • The conditioning recipe transfers from base-to-arm to arm-to-arm coordination, since the RoboTwin experiments adapt mobility-to-body into dual-arm-to-dual-arm conditioning.
  • Real-world deployment is feasible from roughly 100 demonstrations per task, with the model handling at least four subtask stages in a single long-horizon episode.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the cosine-similarity weighting is a zero-shot, language-driven form of view selection; a direct comparison against fixed or learned view-selection heuristics would clarify where the adaptivity actually pays.
  • Because the visualizations of learned weights are retrospective, a natural next experiment is to freeze or randomize the weights while keeping the projection MLPs; if performance stays near the ablated level, the gains are not causally attributable to the similarity mechanism.
  • The mobility-to-body pattern suggests a general design for coupled subsystems in robotics, such as legged locomotion plus arm, two arms, or base plus dexterous hand, where one subsystem's motion prior is extracted by a small pretrained head and injected into whole-body prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes AC-DiT, an end-to-end diffusion transformer for language-conditioned mobile manipulation. It introduces two mechanisms: mobility-to-body conditioning, in which a lightweight mobility action head is pretrained on base-velocity targets to produce latent mobility features that condition the whole-body action head; and perception-aware multimodal adaptation, in which cosine similarity between language features and projected 2D/3D visual features determines per-stage weights for three camera views and the point cloud. The model is evaluated on ManiSkill-HAB (7 tasks), RoboTwin (6 bimanual tasks), and four real-world long-horizon tasks, with comparisons to ACT, DP, 3DP, RDT, and π0. The reported results claim consistent improvements in simulation and real-world settings.

Significance. If the reported results hold, the paper makes a useful contribution: it addresses whole-body coordination by conditioning manipulator actions on base-motion priors and proposes a stage-adaptive fusion of 2D and 3D perception, an underexplored problem in end-to-end mobile manipulation. The paper's strengths include a clearly specified architecture, a two-head training scheme, ablations for each mechanism, and openly discussed limitations and failure modes. The simulation setup is described as using identical observation and action spaces for all baselines (Section 4.1.1), which is a positive design choice. However, the load-bearing evidence currently has internal numerical inconsistencies, an unequal sensor comparison in the real world, and an ablation that does not isolate the adaptive-weighting mechanism; these issues must be resolved before the significance claim is credible.

major comments (4)
  1. [Section 4.1.1, Table 1] The text reports a mean success rate of 48.7% for AC-DiT, but Table 1 lists a mean of 55.6%, and the average of the seven reported per-task values is indeed 55.6. This discrepancy directly affects the headline quantitative claim and must be corrected; the authors should also state which value corresponds to the reported setting.
  2. [Section 4.2 and Appendix B] The real-world comparison is not controlled for sensor input: AC-DiT receives an additional RealSense L515 depth camera, while ACT and π0 receive only RGB from the four Orbbec cameras, as Appendix B explicitly states. Because the point-cloud modality is one of AC-DiT's two architectural claims, the real-world advantage may be due to the extra sensor. Please report results with matched observations (e.g., give baselines the same point cloud, or run AC-DiT without depth) or clearly state that the comparison is system-level with unequal sensors.
  3. [Section 4.3, Table 4] The +2.0 point gain from Exp3 to Exp4 is attributed to PMA, but Exp4 adds the entire PMA block (three projectors, cosine computation, normalization, and reweighting) relative to Exp3. There is no control with fixed or equal weights but identical projectors, so the gain could come from added capacity rather than from adaptivity. The weights depend on cosine similarity to language (Eqs. 1-2), but no experiment varies this signal. Add such a control and report variance or statistical significance; given per-task standard deviations of 1.9-9.4 in Table 1, a 2.0-point gain without variance is not meaningful.
  4. [Section 4.1.1 and Table 1] No statistical tests are reported for the main simulation comparison. With only three runs per task and per-task standard deviations as high as 9.4, the claim of a 'significant margin' over baselines is not supported. Report statistical tests or at least confidence intervals for the mean success rates.
minor comments (6)
  1. [Section 4.1.2, Table 2] The text says DP has an average success rate of only 38.2%, but the table reports 37.9%; please align the text and table.
  2. [Section 4.2, paragraph beginning 'Quantitative Results'] The sentence 'As shown in Figure 3, we compare AC-DiT with ACT...' appears to refer to Table 3, not Figure 3; please correct the cross-reference.
  3. [Figure 1 and Figure 2 captions] The word 'Preception-aware' is a typo for 'Perception-aware'.
  4. [Section 3.4, Eqs. (1)-(2)] Equations (1)-(2) define cosine similarities, but the text says these scores are subsequently normalized to obtain importance weights; the normalization step should be written explicitly in the equations.
  5. [Table 3] The real-world results table is difficult to read because the sub-task columns are not clearly separated; please reformat it so each task and its sub-tasks are visually distinct.
  6. [Appendix C.2] There are typos in the environment names: 'RoboTwen' and 'RobotWin' should both be 'RoboTwin'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons against external baselines, and the only overlapping-author citation (Lift3D) is a component adoption, not a load-bearing premise.

full rationale

The paper does not derive its headline results from its own assumptions; it demonstrates them empirically against external baselines (DP, 3DP, RDT, pi0, ACT) on ManiSkill-HAB, RoboTwin, and real-world tasks. The two proposed mechanisms—mobility-to-body conditioning and perception-aware multimodal adaptation—are evaluated by ablations (Table 4), and the PMA weights are computed by Equations (1)-(2) from inputs (projected visual and language features) rather than fitted to the success-rate outcome, so the adaptive-weighting claim is not self-definitional. The only overlapping-author citation is Lift3D for the 3D tokenizer; this is a component adoption, not a load-bearing premise, and the comparisons do not reduce to it. The absence of a fixed-weight control in the PMA ablation is a potential experimental confound (added projectors and capacity could explain part of the +2.0 gain), but that is a causal-attribution weakness, not circularity, since no fitted parameter is renamed as a prediction and no equation collapses into its own input. No uniqueness theorem or self-citation chain is invoked to force the architecture choice.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical architecture paper rather than a derivation, so the ledger contains hand-chosen hyperparameters and domain assumptions rather than fitted physical constants. No new physical entities are introduced; the latent mobility feature is an internal network representation.

free parameters (3)
  • Observation history window length tau = 2
    Sliding window of two timesteps chosen by hand to give temporal context; affects all compared methods equally in simulation.
  • Action chunk horizon k = 2
    Chunked prediction of two future actions; a design choice that could affect coordination and evaluation outcomes.
  • Denoising steps for mobility latent feature = 5
    The lightweight mobility head's action tokens from its final DiT block at each of five denoising steps are concatenated into F_m; the number of steps is a hand-chosen hyperparameter.
assumptions (4)
  • domain assumption RL-collected demonstrations in ManiSkill-HAB are valid training data for imitation learning of mobile manipulation.
    Section 4.1.1 states trajectories are collected with RL agents (PPO/SAC) and 1000 demos per task; the policy is trained on these as ground truth.
  • domain assumption SigLIP features pretrained on 2D image-text data transfer to 3D point cloud tokens.
    Section 3.2 adopts Lift3D's point-cloud-to-token conversion and a SigLIP encoder with LoRA; transferability is assumed from Lift3D [45] rather than justified in this paper.
  • domain assumption Denoising MSE loss is an adequate objective for generating whole-body actions.
    Section 3.5 uses standard diffusion denoising MSE loss; the paper does not compare alternative action-generation objectives.
  • domain assumption Seventeen real-world trials per task are sufficient to compare methods reliably.
    Section 4.2 evaluates each method for 17 trials and reports no confidence intervals or significance tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation." pith.science (2026). https://pith.science/paper/OWOIWDFA

@misc{pith2026250701961,
  author       = {Pith},
  title        = {Pith review of: AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWOIWDFA}},
  note         = {Machine review of arXiv:2507.01961}
}
read the original abstract

Recently, mobile manipulation has attracted increasing attention for enabling language-conditioned robotic control in household tasks. However, existing methods still face challenges in coordinating mobile base and manipulator, primarily due to two limitations. On the one hand, they fail to explicitly model the influence of the mobile base on manipulator control, which easily leads to error accumulation under high degrees of freedom. On the other hand, they treat the entire mobile manipulation process with the same visual observation modality (e.g., either all 2D or all 3D), overlooking the distinct multimodal perception requirements at different stages during mobile manipulation. To address this, we propose the Adaptive Coordination Diffusion Transformer (AC-DiT), which enhances mobile base and manipulator coordination for end-to-end mobile manipulation. First, since the motion of the mobile base directly influences the manipulator's actions, we introduce a mobility-to-body conditioning mechanism that guides the model to first extract base motion representations, which are then used as context prior for predicting whole-body actions. This enables whole-body control that accounts for the potential impact of the mobile base's motion. Second, to meet the perception requirements at different stages of mobile manipulation, we design a perception-aware multimodal conditioning strategy that dynamically adjusts the fusion weights between various 2D visual images and 3D point clouds, yielding visual features tailored to the current perceptual needs. This allows the model to, for example, adaptively rely more on 2D inputs when semantic information is crucial for action prediction, while placing greater emphasis on 3D geometric information when precise spatial understanding is required. We validate AC-DiT through extensive experiments on both simulated and real-world mobile manipulation tasks.

Figures

Figures reproduced from arXiv: 2507.01961 by the authors.

Figure 1
Figure 1. Overview of AC-DiT. The proposed end-to-end mobile manipulation framework enhances the coordination between the mobile base and the manipulator by introducing two key mechanisms: mobile-to-body conditioning and perception-aware multimodal adaptation. The former enables action prediction that conditioning on how upcoming mobile base movements may affect manipulator control, thereby reducing error accumulation. The la… view at source ↗
Figure 2
Figure 2. AC-DiT framework.We first train the modules in the grey-shaded region under the supervision of mobile base actions, allowing the lightweight mobility action head to learn to extract latent mobility features. After this, we optimize the entire AC-DiT model, enabling the mobile manipulation action head to predict both mobile base and manipulator actions. With the Mobility-to￾Body Conditioning mechanism, this action he… view at source ↗
Figure 3
Figure 3. Robot execution visualization of 7 tasks in mobile simulator ManiSkill-HAB and 6 tasks in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 3
Figure 3. Figure 3: We adopt the same baselines and evaluation metrics as described in Section 4.1.1, in which [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Robot execution progress of AC-DiT in four real-world tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Effectiveness of Perception-aware Multimodal [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Robot hardware configuration and real-world assets. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Robot execution progress of AC-DiT in real-world tasks. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Observation examples in real-world tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Robot execution progress of AC-DiT in seven MSHab simulation tasks. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Observation examples in MSHab and RoboTwin simulations. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Robot execution progress of AC-DiT in seven RoboTwin simulation tasks. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Visualization of Perception-aware Multimodal Adaption mechanism on a whole episode. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation

    cs.RO 2026-08 conditional novelty 6.0 of 10

    Adding a panoramic camera feed to a vision-language-action policy raises end-to-end success on four real-world mobile two-arm tasks from 30% to 73%.

  2. InCoM: Intent-Driven Perception and Structured Coordination for Mobile Manipulation

    cs.RO 2026-02 unverdicted novelty 6.0 of 10

    InCoM reports 23–28 percentage-point success-rate gains in mobile manipulation benchmarks by dynamically reweighting multi-scale perception via inferred motion intent and decoupling base-arm action generation with flo...

Reference graph

Works this paper leans on

55 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning , 2022

    work page 2022

  2. [2]

    Wildlma: Long horizon loco-manipulation in the wild

    Ri-Zhao Qiu, Yuchen Song, Xuanbin Peng, Sai Aneesh Suryadevara, Ge Yang, Minghuan Liu, Mazeyu Ji, Chengzhe Jia, Ruihan Yang, Xueyan Zou, et al. Wildlma: Long horizon loco-manipulation in the wild. arXiv preprint arXiv:2411.15131, 2024

    arXiv 2024

  3. [3]

    Robomatrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open-world

    Weixin Mao, Weiheng Zhong, Zhou Jiang, Dong Fang, Zhongyue Zhang, Zihan Lan, Fan Jia, Tiancai Wang, Haoqiang Fan, and Osamu Yoshie. Robomatrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open-world. arXiv preprint arXiv:2412.00171, 2024

    arXiv 2024

  4. [4]

    Ok-robot: What really matters in integrating open-knowledge models for robotics

    Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401.12202, 2024

    arXiv 2024

  5. [5]

    Homer- obot: Open-vocabulary mobile manipulation

    Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, et al. Homer- obot: Open-vocabulary mobile manipulation. In 7th Annual Conference on Robot Learning , 2023

    work page 2023

  6. [6]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    arXiv 2023

  7. [7]

    Mobile aloha: Learning bimanual mobile manipu- lation using low-cost whole-body teleoperation

    Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipu- lation using low-cost whole-body teleoperation. In 8th Annual Conference on Robot Learning , 2024

    work page 2024

  8. [8]

    M 2 diffuser: Diffusion-based trajectory optimization for mobile manipulation in 3d scenes

    Sixu Yan, Zeyu Zhang, Muzhi Han, Zaijin Wang, Qi Xie, Zhitian Li, Zhehan Li, Hangxin Liu, Xinggang Wang, and Song-Chun Zhu. M 2 diffuser: Diffusion-based trajectory optimization for mobile manipulation in 3d scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    work page 2025

Show all 55 references
  1. [9]

    Behavior robot suite: Streamlining real-world whole- body manipulation for everyday household activities

    Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, and Li Fei-Fei. Behavior robot suite: Streamlining real-world whole- body manipulation for everyday household activities. arXiv preprint arXiv:2503.05652, 2025

    2025 arXiv

  2. [10]

    Error-aware imitation learning from teleoperation data for mobile manipulation

    Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, and Roberto Martín-Martín. Error-aware imitation learning from teleoperation data for mobile manipulation. In 5th Annual Conference on Robot Learning , 2022

    2022

  3. [11]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024. 10

    2024 arXiv

  4. [12]

    Maniskill-hab: A benchmark for low-level manipulation in home rearrangement tasks

    Arth Shukla, Stone Tao, and Hao Su. Maniskill-hab: A benchmark for low-level manipulation in home rearrangement tasks. arXiv preprint arXiv:2412.13211, 2024

    2024 arXiv

  5. [13]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , page 02783649241273668, 2023

    2023

  6. [14]

    3d diffusion policy

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. arXiv e-prints, pages arXiv–2403, 2024

    2024

  7. [15]

    π0: A vision-language-action flow model for general robot control, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control, 2024. URL https://arxiv. org/abs/2410.24164, 2024

    2024 arXiv

  8. [16]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  9. [17]

    Gpt- 4v (ision) for robotics: Multimodal task planning from human demonstration

    Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt- 4v (ision) for robotics: Multimodal task planning from human demonstration. IEEE Robotics and Automation Letters, 2024

    2024

  10. [18]

    Spatial-language attention policies for efficient robot learning

    Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Yonatan Bisk, and Chris Paxton. Spatial-language attention policies for efficient robot learning. arXiv preprint arXiv:2304.11235, 2023

    2023 arXiv

  11. [19]

    Open-television: Teleop- eration with immersive active visual feedback

    Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Xiaolong Wang. Open-television: Teleop- eration with immersive active visual feedback. arXiv preprint arXiv:2407.01512, 2024

    2024 arXiv

  12. [20]

    Mobile-television: Predictive motion priors for humanoid whole-body control

    Chenhao Lu, Xuxin Cheng, Jialong Li, Shiqi Yang, Mazeyu Ji, Chengjing Yuan, Ge Yang, Sha Yi, and Xiaolong Wang. Mobile-television: Predictive motion priors for humanoid whole-body control. arXiv preprint arXiv:2412.07773, 2024

    2024 arXiv

  13. [21]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018

    2018

  14. [22]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  15. [23]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021

    2021 arXiv

  16. [24]

    Robotic grasping using deep reinforcement learning

    Shirin Joshi, Sulabh Kumra, and Ferat Sahin. Robotic grasping using deep reinforcement learning. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 1461–1466. IEEE, 2020

    2020

  17. [25]

    Learning dexterous in-hand manipulation

    OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3–20, 2020

    2020

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  19. [27]

    Anygrasp: Robust and efficient grasp perception in spatial and temporal domains

    Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 39(5):3929–3945, 2023

    2023

  20. [28]

    Adaptive mobile manipulation for articulated objects in the open world

    Haoyu Xiong, Russell Mendonca, Kenneth Shaw, and Deepak Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint arXiv:2401.14403, 2024. 11

    2024 arXiv

  21. [29]

    Harmonic mobile manipulation

    Ruihan Yang, Yejin Kim, Rose Hendrix, Aniruddha Kembhavi, Xiaolong Wang, and Kiana Ehsani. Harmonic mobile manipulation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3658–3665. IEEE, 2024

    2024

  22. [30]

    Spin: Si- multaneous perception, interaction and navigation

    Shagun Uppal, Ananye Agarwal, Haoyu Xiong, Kenneth Shaw, and Deepak Pathak. Spin: Si- multaneous perception, interaction and navigation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18133–18142. IEEE, 2024

    2024

  23. [31]

    Openvla: An open-source vision-language-action model

    Moo Jsn Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  24. [32]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  25. [33]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation

    Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  26. [34]

    Robomamba: Multimodal state space model for efficient robot reasoning and manipulation

    Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. arXiv preprint arXiv:2406.04339, 2024

    2024 arXiv

  27. [35]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  28. [36]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023

    2023 arXiv

  29. [37]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

    arXiv

  30. [38]

    Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models

    Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. In 2024 IEEE/RSJ International Confer- ence on In...

    2024

  31. [39]

    3d diffuser actor: Policy diffusion with 3d scene representations

    Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885, 2024

    2024 arXiv

  32. [40]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  33. [41]

    Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation

    Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024

    2024 arXiv

  34. [42]

    Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression

    Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vla: Scaling robot foundation models via unified diffusion and autoregression. arXiv preprint arXiv:2412.03293, 2024

    2024 arXiv

  35. [43]

    Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation

    Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. IEEE Robotics and Automation Letters , 2025. 12

    2025

  36. [44]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  37. [45]

    Lift3d foundation policy: Lift- ing 2d large-scale pretrained models for robust 3d robotic manipulation

    Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, et al. Lift3d foundation policy: Lift- ing 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.18623, 2024

    2024 arXiv

  38. [46]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  39. [47]

    Sapien: A simulated part-based interactive environ- ment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environ- ment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages ...

    2020

  40. [48]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing ...

    2021

  41. [49]

    The ycb object and model set: Towards common benchmarks for manipulation research

    Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR) , pages 510–517. IEEE, 2015

    2015

  42. [50]

    Robotwin: Dual-arm robot benchmark with generative digital twins (early version)

    Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (early version). arXiv preprint arXiv:2409.02920, 2024

    2024 arXiv

  43. [51]

    Cucumber in basket

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4195–4205, 2023. 13 Appendix A Robot Hardware Details. In this section, we provide a detailed description of the rob...

    2023

  44. [52]

    Cucumber in basket. The robotic setup includes a primary worktable in front of the robot holding a cucumber and a secondary table on the robot’s left side with a basket for object placement, where the robot starts at an initial distance from the primary worktable with its arm ...

  45. [53]

    The robotic workspace comprises a primary worktable positioned in front of the robot, equipped with two bread items and a basket, and a secondary left-side table initially empty

    Store bread. The robotic workspace comprises a primary worktable positioned in front of the robot, equipped with two bread items and a basket, and a secondary left-side table initially empty. 15 Store BreadsHang Towel Cucumber in BasketClean Table Lift-wrist Exterior Right-wri...

  46. [54]

    Hang towel. The robotic task environment consists of a workdbench positioned in front of the robot, bearing a towel as the target object, and a hanging rack located on the robot’s left lateral side. The robot initiates the task at an initial pose in front of the workbench with...

  47. [55]

    Mobility-to-Body

    Clean table. The robotic workspace features a workbench equipped with four waste objects: a Sprite bottle, a Cestbon bottle, orange peel, and a crumpled tissue. A large bowl and two randomly positioned waste objects are initially placed on the workbench’s left side, with the r...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.