Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReasonManip shows that a 7B multimodal model can drive direct robot manipulation by predicting each next gripper pose in language after being fine-tuned on only 65 teacher dialogues and reinforced with GRPO in simulation.

desk verdict The paper has a genuinely useful idea in the axis-based pose representation and a striking data-efficiency result, but the central claim that system-2 reasoning drives performance is not actually tested. read the letter →

arxiv 2505.12744 v1 pith:3XU5UE5F submitted 2025-05-19 cs.AI

Weiliang Tang , Dong Jing , Jia-Hui Pan , Zhiwu Lu , Yun-Hui Liu , Li Erran Li , Mingyu Ding , Chi-Wing Fu This is my paper · ORCID
classification cs.AI
keywords robotmanipulationlargemultimodalmodelsreasoningGRPOreinforcementlearningsim-to-realtransferspatialvision-language-action
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large multimodal model can control a robot manipulator directly by reasoning out each next gripper pose in natural language, rather than by emitting actions through a separate decoder or action head. To make that possible, it reformulates manipulation as multi-round dialogues in which the model receives quantitative scene state — each object part's center, bounding box, and three PCA-derived orientation axes — and must output the next gripper position, orientation, and open/close command. The resulting 7B model, ReasonManip, is fine-tuned on just 65 human-guided dialogues generated by a 72B teacher, then improved by GRPO trial-and-error in the SIMPLER simulator. The authors report zero-shot transfer to unseen MetaWorld tasks and to a real UR5 robot, with success rates that beat RT-1-X, OpenVLA, and pi0 and that barely change when the camera viewpoint moves. If right, this means robot manipulation can become data-efficient, interpretable, and broadly generalizable when posed as text-based spatial reasoning, provided an external perception layer faithfully supplies the numbers the model reasons over.

What carries the argument

The load-bearing object is the axis-based rotation representation; the load-bearing mechanism is the two-stage training loop. Each object part is encoded as a 12-tuple: Cartesian center $(x,y,z)$, bounding-box dimensions $(l,w,h)$, and three PCA-derived orthogonal unit vectors — the longitudinal axis (largest eigenvector), the normal axis (smallest eigenvector), and the binormal axis given by their cross product. Gripper poses are described the same way, with the rotation matrix reconstructed from the axis vectors by a fixed formula, which turns orientation reasoning from opaque Euler angles into vector arithmetic the model can perform in text. The training loop first supervises a 7B LMM on 65 multi-round dialogues produced by a 72B teacher under human guidance, then runs GRPO inside SIMPLER with binary task success as reward, rewarding text that checks collisions, detects misalignment, and corrects poses before every executed action.

What would settle it

Run ReasonManip on the same SIMPLER tasks while corrupting only the supplied scene state — add Gaussian noise to object positions, or remove part of an object's point cloud so its PCA axes become unstable — while leaving the visual observation unchanged. If success collapses in proportion to the corruption, the policy is carried by the external numbers rather than by the model's spatial reasoning; the paper's own Appendix F names incomplete point clouds as a failure condition. A complementary check is to attempt tasks on symmetric objects such as a perfect cube or sphere, where the PCA axis representation is ambiguous or degenerate, and count the resulting failures.

Watch

Extended reading notes

Core claim

ReasonManip establishes that next-goal gripper poses can be produced as plain text through explicit, step-by-step spatial reasoning, provided the spatial action space is re-encoded from Euler angles into three orthogonal unit vectors (longitudinal, binormal, and normal axes) computed from point-cloud PCA of each object part. On this formulation, high-level task decomposition, mid-level goal analysis, and low-level pose estimation all happen in a single dialogue pass in one unified language format. Fine-tuned on 65 teacher dialogues and then refined with GRPO inside SIMPLER, the 7B model reaches an average 45.4% success across seven SIMPLER tasks, transfers zero-shot to unseen MetaWorld environments (96.7% on grasping, where RT-1-X, Octo-base, and OpenVLA all score 0%), keeps near-identical success when the camera view changes, and drives a real UR5 arm at 40–50% success on three tasks, outperforming even real-world-fine-tuned OpenVLA and pi0. The authors attribute this generalization to the mathematical, appearance-invariant form of the task and to GRPO, which they argue mitigates SFT overfitting and induces written reasoning behaviors such as collision checks, failure detection, and self-correction before each action.

Load-bearing premise

The load-bearing premise is that accurate, complete quantitative scene state — every object part's position, bounding-box size, and three PCA-derived orientation axes, plus the gripper pose — is supplied to the model in text at every dialogue turn; the paper itself (Appendix F) acknowledges failure when point clouds are incomplete, and if this perception layer is noisy or wrong, the reasoning and the executed actions fail regardless of the language model's quality.

Editorial extensions

If this is right

  • A single language interface can carry all levels of control: task decomposition, goal analysis, and concrete pose estimation happen in one reasoning pass, so no separate planner or action decoder is required.
  • Manipulation policies can be trained with far less data — 65 dialogues plus simulator reinforcement — than the millions of demonstrations typically used for VLA models.
  • The axis representation is itself a performance lever: reformatting all orientations as Euler angles in the same training setup drops lift-coke-can success from 96.7% to 42.0% in the paper's comparison.
  • Policies that reason over quantitative state rather than raw appearance transfer zero-shot across simulators and to a real robot, and their success is nearly unchanged under large camera-view changes.
  • Every executed action is accompanied by a written rationale, so failures can be traced back to a specific reasoning step rather than an opaque action tensor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The practical ceiling of this paradigm sits in the perception front-end, not the reasoning model: because the policy receives, rather than computes, the quantitative state, the reported successes presuppose accurate segmentation and PCA, so point-cloud completion or better pose estimation might buy more robot-level robustness than a larger language model would.
  • The formulation is naturally limited to tasks decomposable into a short sequence of discrete absolute poses, such as pick, place, push, and close; continuous or trajectory-sensitive skills such as wiping or pouring would need the hybrid scheme the authors sketch, switching between deliberate next-goal reasoning and high-frequency direct generation.
  • Since GRPO is run per-task rather than jointly (the authors note severe task conflict with only 65 samples), a meaningful stress test would be training on all tasks together and measuring how much of the out-of-distribution transfer survives.
  • An independent ablation would isolate the sources of the gain: applying the same dialogue-and-axis format to an open 7B VLA without GRPO would show how much of the sim-to-real transfer comes from the representation itself versus from reinforcement-driven exploration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReasonManip, a 7B large multimodal model that performs tabletop robot manipulation by predicting next-goal gripper poses in natural language using a novel axis-based rotation representation. The training pipeline consists of (i) collecting 65 multi-round reasoning dialogues from Qwen2.5-VL-72B with human guidance, (ii) supervised fine-tuning of Qwen2.5-VL-7B, and (iii) Group Relative Policy Optimization (GRPO) inside the SIMPLER simulator with a binary success reward. The authors claim that the resulting model exhibits system-2 reasoning behaviors, achieves high out-of-distribution generalization to MetaWorld, transfers to a real UR5 robot, and offers transparent interpretability. Experiments compare ReasonManip with RT-1-X, Octo, OpenVLA, and pi0 on SIMPLER, MetaWorld, and a real-world setup, reporting success rates that generally favor ReasonManip, alongside a small ablation of the axis representation versus Euler angles.

Significance. If the claims were fully supported, the paper would introduce a compelling paradigm: using language as a unified action and reasoning interface for manipulation, with extreme data efficiency (65 demonstrations), zero-shot transfer across simulators, and interpretable step-by-step control. The axis-based rotation representation and the GRPO-in-the-loop training recipe are potentially valuable. The paper also honestly states some limitations in Appendix F. However, the central attribution of the reported success to system-2 reasoning is currently unsubstantiated: the comparisons are confounded by privileged numeric state input, the reasoning tokens are never ablated, and the OOD claims are weakened by per-task GRPO training. The ideas are interesting and the topic is timely, but the evidence as presented does not yet support the strong conclusions.

major comments (4)
  1. [Sec. 3.2, Tables 1-3] The performance comparison against VLA baselines is confounded because ReasonManip receives privileged quantitative scene information (object centers, bounding-box dimensions, PCA-derived axes, and gripper pose) as text at every dialogue turn, whereas RT-1-X, Octo, OpenVLA, and pi0 receive only images. The reported success-rate advantages therefore cannot be attributed to reasoning or the language interface; a model that simply maps state vectors to goal poses might achieve comparable results. The paper should include a controlled comparison where baselines receive an equivalent state representation (e.g., as additional input tokens or via the same perception module), or an ablation in which ReasonManip is evaluated without textual scene information.
  2. [Sec. 3.4, Sec. 4.5, Eq. (3)] The central claim that system-2 reasoning drives performance is not supported by causal evidence. The SFT loss in Eq. (2) trains on teacher-generated reasoning, and the GRPO loss in Eq. (3) rewards only binary task success, never the quality or causal role of the reasoning text. The model could learn to emit correct actions while producing fluent but post-hoc rationalizations. The qualitative examples in Sec. 4.5 and Fig. 8 illustrate failure detection and self-correction, but they do not show that the reasoning tokens are load-bearing. An ablation that removes, corrupts, or decouples the <think> tokens—for instance, training an action-only variant or replacing reasoning text with fixed filler during inference—should be performed. Without such an ablation, the paper's stated three advantages (OOD generalization, sim-to-real transfer, interpretability) cannot be causally credited to the reasoning process.
  3. [Appendix A, Table 2] The out-of-distribution generalization claim is weakened by the fact that GRPO is trained per task. Appendix A states: 'We use the GRPO to train for each of the specific tasks, since we observe severe task conflict based on the limited quantity of SFT training samples.' The MetaWorld tasks in Table 2 are described as 'resembling those in SIMPLER' (Grasp, Move, Stack, Close), i.e., they are the same task types across different simulators and embodiments, not unseen task categories. The abstract's wording 'exceptional generalizability to out-of-distribution environments, objects, and tasks' is therefore not supported for the 'tasks' component. Please clarify what is claimed to generalize and provide evidence for generalization to genuinely novel task types, or adjust the claims accordingly.
  4. [Tables 1-4] None of the reported success rates include confidence intervals or statistical significance tests, which matters because several headline differences are small relative to the trial counts (50 trials per SIMPLER task, 30 per MetaWorld task, 10 per real-world task). For example, Table 2 shows Move improving from 23.3% to 30.0% and Close from 20% to 40% after GRPO; with 30 trials, these differences are within plausible sampling noise. Similarly, Table 1 reports many values in the 30-45% range with no error bars. The authors should report per-trial variability (e.g., bootstrap confidence intervals over episodes) and, where possible, the number of independent evaluation seeds.
minor comments (5)
  1. [Sec. 2] There are naming typos: 'Deekseek-R1' should be 'DeepSeek-R1' and 'GPRO' should be 'GRPO'.
  2. [Fig. 3] The caption of Figure 3 is incomplete; the bottom-left subplots showing Euler angles are not described in the caption, and the figure contains dense inline text that is hard to read. Please expand the caption and simplify the figure.
  3. [Eq. (3)] The notation in Eq. (3) is inconsistent: 'at.i' appears in the policy ratio but 'at,i' in the loss sum; also the definition of clip(·) is written in a compressed way. Please clean up the typography.
  4. [Appendix A] Please report the GRPO hyperparameters more completely, including the group size K, the number of update epochs per task, and whether rewards include any shaping beyond the binary success flag.
  5. [Sec. 4.4] The real-world experiments would benefit from reporting the accuracy of the support-query segmentation module, since the method's failures are attributed in Appendix F to incomplete point clouds. Without this, it is unclear whether the 40-50% real-world success rates are limited by perception, reasoning, or control.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the pose predictions are trained against external simulator success and tested on unseen environments, with only auxiliary self-citations.

full rationale

ReasonManip's central chain is empirical rather than definitional. The teacher dialogues are produced by an external 72B LMM with human guidance (Sec. 3.3), SFT is cross-entropy on those dialogues (Eq. 2), and GRPO uses a binary task-success reward from the SIMPLER simulator (Eq. 3). The output next-goal gripper pose is parsed from the model's answer and executed; it is not by construction identical to the scene-state input, and the MetaWorld/real-robot evaluations use environments not seen during training, so the claimed generalization is an external test rather than a restatement of the training objective. The axis-representation advantage is directly tested against Euler (Table 5). The only self-citations ([10], [36], [37]) support auxiliary components — a segmentation network and a motivation about open-world segmentation being unsatisfactory — and do not carry the core claim. The absence of a reasoning-token ablation (or a reward that specifically scores reasoning quality) leaves the attribution of success to 'system-2 reasoning' as an open experimental question, but it is not circular: the GRPO reward does not contain the reasoning text as a target, and the reasoning tokens are not fitted to the evaluation metric. Similarly, Appx. F's stated failure mode under incomplete point clouds is an external dependency, not a fitted-input circularity. No step reduces to its own input by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted constants are required for the central claim; the method depends on standard linear algebra, the availability of accurate scene state, and the sufficiency of a binary task-success reward. The axis representation is a change of coordinates, not an invented entity.

assumptions (5)
  • domain assumption Tabletop manipulation tasks can be decomposed into a sequence of discrete next-goal gripper end poses.
    This is the core task formulation in Sec. 3.2; the authors themselves exclude curvy trajectories such as wiping a bowl (Appx. F).
  • domain assumption Accurate quantitative scene information (positions, dimensions, axes, gripper state) is available at each step.
    The model input is this text state (Sec. 3.2, Eq. 1); real-world deployment depends on the segmentation module of Sec. 3.5, and Appx. F notes failures when point clouds are incomplete.
  • domain assumption The binary simulator success signal is a sufficient reward for learning policies that generalize.
    GRPO uses only this reward (Eq. 3); the paper claims it improves OOD performance, but this is not proven.
  • standard math Gripper orientation can be reconstructed from predicted longitudinal and binormal axes via the standard orthonormal-basis formula.
    Eq. 1 reconstructs the rotation matrix from two orthonormal axes; this is standard linear algebra and is valid if the predicted vectors are properly normalized and orthogonal.
  • domain assumption A 7B LMM can learn spatial reasoning from a 72B teacher's dialogues and GRPO.
    The entire training pipeline (Sec. 3.4) assumes the smaller model can absorb the teacher's reasoning; the paper provides empirical support but no guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation." pith.science (2026). https://pith.science/paper/3XU5UE5F

@misc{pith2026250512744,
  author       = {Pith},
  title        = {Pith review of: Incentivizing Multimodal Reasoning in Large Models for Direct Robot Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XU5UE5F}},
  note         = {Machine review of arXiv:2505.12744}
}
read the original abstract

Recent Large Multimodal Models have demonstrated remarkable reasoning capabilities, especially in solving complex mathematical problems and realizing accurate spatial perception. Our key insight is that these emerging abilities can naturally extend to robotic manipulation by enabling LMMs to directly infer the next goal in language via reasoning, rather than relying on a separate action head. However, this paradigm meets two main challenges: i) How to make LMMs understand the spatial action space, and ii) How to fully exploit the reasoning capacity of LMMs in solving these tasks. To tackle the former challenge, we propose a novel task formulation, which inputs the current states of object parts and the gripper, and reformulates rotation by a new axis representation instead of traditional Euler angles. This representation is more compatible with spatial reasoning and easier to interpret within a unified language space. For the latter challenge, we design a pipeline to utilize cutting-edge LMMs to generate a small but high-quality reasoning dataset of multi-round dialogues that successfully solve manipulation tasks for supervised fine-tuning. Then, we perform reinforcement learning by trial-and-error interactions in simulation to further enhance the model's reasoning abilities for robotic manipulation. Our resulting reasoning model built upon a 7B backbone, named ReasonManip, demonstrates three notable advantages driven by its system-2 level reasoning capabilities: i) exceptional generalizability to out-of-distribution environments, objects, and tasks; ii) inherent sim-to-real transfer ability enabled by the unified language representation shared across domains; iii) transparent interpretability connecting high-level reasoning and low-level control. Extensive experiments demonstrate the effectiveness of the proposed paradigm and its potential to advance LMM-driven robotic manipulation.

Figures

Figures reproduced from arXiv: 2505.12744 by the authors.

Figure 1
Figure 1. An illustration of ReasonManip that solves robot manipulation tasks by next-goal prediction via unified [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall pipeline of our method. On the first stage, we harness the advanced LMM-72B to generate robot [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example of the axis representation for the orientation of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The illustration of our multi-round conversation collection pipeline. The predicted gripper pose is executed in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the environment comparisons of the SIMPLER and the MetaWorld [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Sequence of our method’s execution in the real-world environment for the three tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Example of conversation demonstrating the system-2 reasoning ability. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Illustration of the real-world experiment settings. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Examples that LMM fails to generate correct gripper pose under Euler representation. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Summary of multi-round conversations for achieving "stack green cube on yellow cube". [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Conversation for "Stack the green cube onto the yellow cube" - Round 1 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Conversation for "Stack the green cube onto the yellow cube" - Round 2 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Conversation for "Stack the green cube onto the yellow cube" - Round 3 [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Conversation for "Stack the green cube onto the yellow cube" - Round 4 [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Conversation for "Stack the green cube onto the yellow cube" - Round 5 [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixture of Horizons in Action Chunking

    cs.RO 2025-11 conditional novelty 6.0 of 10

    A gated mixture of multiple action-chunk horizons in a shared full-attention transformer improves VLA manipulation success and enables consensus-based early stopping.

  2. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    VLAs fail most counterfactual instructions because vision shortcuts dominate language; the new LIBERO-CF benchmark quantifies this, and CAG, an inference-time action mixer, improves grounding and success.

Reference graph

Works this paper leans on

50 extracted references · 4 canonical work pages · cited by 2 Pith papers

  1. [1]

    Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  2. [2]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    arXiv 2025

  3. [3]

    pi_0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi_0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  4. [4]

    Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

  5. [5]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  6. [6]

    Igor: Image-goal representations are the atomic control units for foundation models in embodied ai.arXiv preprint arXiv:2411.00785, 2024

    Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai.arXiv preprint arXiv:2411.00785, 2024

    arXiv 2024

  7. [7]

    Moto: Latent motion token as the bridging language for robot manipulation.arXiv preprint arXiv:2412.04445, 2024

    Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for robot manipulation.arXiv preprint arXiv:2412.04445, 2024

    arXiv 2024

  8. [8]

    Diffusion policy: Visuomotor policy learning via action diffusion.arXiv preprint arXiv:2303.04137, 2023

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.arXiv preprint arXiv:2303.04137, 2023

    arXiv 2023

Show all 50 references
  1. [9]

    Dynamo: In-domain dynamics pretraining for visuo.Motor Control, 2024

    ZJ Cui, H Pan, A Iyer, S Haldar, and L Pinto. Dynamo: In-domain dynamics pretraining for visuo.Motor Control, 2024

    2024

  2. [10]

    Fineclip: Self-distilled region-based clip for better fine-grained understanding.Advances in Neural Information Processing Systems, 2024

    Zhiwu Lu Dong Jing, Xiaolong He et al. Fineclip: Self-distilled region-based clip for better fine-grained understanding.Advances in Neural Information Processing Systems, 2024

    2024

  3. [11]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  4. [12]

    Bridge data: Boosting generalization of robotic skills with cross-domain datasets.arXiv preprint arXiv:2109.13396, 2021

    Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets.arXiv preprint arXiv:2109.13396, 2021

    2021 arXiv

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  7. [15]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  8. [16]

    Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  9. [17]

    An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023. 10 APREPRINT- SEPTEMBER16, 2025

    2023 arXiv

  10. [18]

    V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  11. [19]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    2024 arXiv

  12. [20]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  13. [21]

    macmillan, 2011

    Daniel Kahneman.Thinking, fast and slow. macmillan, 2011

    2011

  14. [22]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla...

    2024 arXiv

  15. [23]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

  16. [24]

    Vision-language foundation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators.arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  17. [25]

    Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation.arXiv ...

    2024 arXiv

  18. [26]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2023

  19. [27]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  20. [28]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  21. [29]

    Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  22. [30]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  23. [31]

    Open x-embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864, 2023

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models.arXiv preprint arXiv:2310.08864, 2023

    arXiv

  24. [32]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  25. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models.arXiv preprint arXiv:2502.19417, 2025

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models.arXiv preprint arXiv:2502.19417, 2025

    2025 arXiv

  27. [35]

    Computing euler angles from a rotation matrix.Retrieved on August, 6(2000):39–63, 1999

    Gregory G Slabaugh. Computing euler angles from a rotation matrix.Retrieved on August, 6(2000):39–63, 1999. 11 APREPRINT- SEPTEMBER16, 2025

    2000

  28. [36]

    Geomanip: Geometric constraints as general interfaces for robot manipulation.arXiv preprint arXiv:2501.09783, 2025

    Weiliang Tang, Jia-Hui Pan, Yun-Hui Liu, Masayoshi Tomizuka, Li Erran Li, Chi-Wing Fu, and Mingyu Ding. Geomanip: Geometric constraints as general interfaces for robot manipulation.arXiv preprint arXiv:2501.09783, 2025

    2025 arXiv

  29. [37]

    Overcoming support dilution for robust few-shot semantic segmentation.arXiv preprint arXiv:2501.13529, 2025

    Weiling Tang, Biqi Yang, Pheng-Ann Heng, Yun-Hui Liu, and Chi-Wing Fu. Overcoming support dilution for robust few-shot semantic segmentation.arXiv preprint arXiv:2501.13529, 2025

    2025 arXiv

  30. [38]

    Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [40]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, An- dre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723–1736. PMLR, 2023

    2023

  33. [41]

    Dart-llm: Dependency-aware multi-robot task decomposition and execution using large language models.arXiv preprint arXiv:2411.09022, 2024

    Yongdong Wang, Runze Xiao, Jun Younes Louhi Kasahara, Ryosuke Yajima, Keiji Nagatani, Atsushi Yamashita, and Hajime Asama. Dart-llm: Dependency-aware multi-robot task decomposition and execution using large language models.arXiv preprint arXiv:2411.09022, 2024

    2024 arXiv

  34. [42]

    Q-learning.Machine learning, 8:279–292, 1992

    Christopher JCH Watkins and Peter Dayan. Q-learning.Machine learning, 8:279–292, 1992

    1992

  35. [43]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  36. [44]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  37. [45]

    Deepcritic: Deliberate critique with large language models.arXiv preprint arXiv:2505.00662, 2025

    Wenkai Yang, Jingwen Chen, Yankai Lin, and Ji-Rong Wen. Deepcritic: Deliberate critique with large language models.arXiv preprint arXiv:2505.00662, 2025

    2025 arXiv

  38. [46]

    Latent action pretraining from videos.arXiv preprint arXiv:2410.11758, 2024

    Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos.arXiv preprint arXiv:2410.11758, 2024

    2024 arXiv

  39. [47]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

    2025 arXiv

  40. [48]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. InConference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  41. [49]

    Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning.arXiv preprint arXiv:2407.08693, 2024

    2024 arXiv

  42. [50]

    put on" task,

    Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model.arXiv preprint arXiv:2403.09631, 2024. 12 APREPRINT- SEPTEMBER16, 2025 A Implementation Details of the Experiments Tr...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.