Pith. sign in

REVIEW 3 major objections 4 minor 58 references

HumanCLAW claims that current vision-language models fail at embodied action because they lack embodied self-awareness, not because they cannot perceive targets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new full-body benchmark shows that current VLMs can recognize targets but cannot reliably tell where their own body is, whether it arrived, or whether it collided; the best solves only 16.8% of episodes.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful benchmark and a credible negative result, but the 'self-awareness' diagnosis goes beyond what the half-physics validation supports. the 3 major comments →

arxiv 2607.27180 v2 pith:FTGAKXJR submitted 2026-07-29 cs.CV cs.RO

HumanCLAW: Can Vision-Language Models Act Through a Body?

Li Siyao , Jiawei Gu , Shuai Liu , Kairui Hu , Zekun Li , Linjie Li , Chengcheng Tang , Po-Chen Wu
show 10 more authors
Ivan Shugurov Lingni Ma Michael Zollhoefer Sizhe An Abhay Mittal Amy Zhao Ranjay Krishna Manling Li Ziwei Liu Chuan Guo
This is my paper
classification cs.CV cs.RO
keywords embodied self-awarenessvision-language modelsaction intelligencehalf-physics simulationhumanoid motion generationfind-navigate-interact benchmarkegocentric spatial reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HumanCLAW is an evaluation framework and benchmark for whether vision-language models (VLMs) can act through a physical body. The paper's claim is that when low-level motor execution is removed from the loop, current frontier VLMs still fail an indoor find-navigate-interact task, and fail specifically because they lose track of their own body: where it is, whether it has reached the goal, and whether it has collided with the world. Across 1,218 episodes in 41 scenes, nine off-the-shelf VLMs top out at 16.8 percent full-task success, with perception (seeing the target) nearly solved. The diagnosis is a missing faculty the paper calls embodied self-awareness, and because motor errors are factored out by construction, the failures are attributed to the decision maker. The framework is built on half-physics simulation, a skill-conditioned motion generator, and a verifier that keeps the decision loop physically grounded.

Core claim

The central claim is that today's VLMs reason about the scene but not about the body they control. With balance and motor tracking removed by the half-physics simulator, the measured success rates—best 16.8 percent for full episodes, 42.4 percent for navigation alone—cannot be blamed on locomotion. The paper then traces the failures with an automated root-cause analysis: once a target is rendered in the egocentric view it is almost always recognized, but 68 percent of found targets are never reached, and among those reached, 71 percent of sits fail. The dominant error modes are stopping while still far, walking on after arrival, sitting into empty air, and colliding with visible obstacles wh

What carries the argument

Half-physics simulation is the load-bearing mechanism: the world follows rigid-body physics—gravity, contact, friction, movable objects—while the humanoid body is driven by equivalent kinematic velocities from a skill-conditioned motion generator, with no joint torques, so balance and motor-tracking failures cannot occur. Around it, the harness exposes a fixed set of atomic whole-body skills (walk, turn, side-step, step back, climb, sit, stop) with continuous parameters in a text/JSON interface; a skill-specific verifier rejects unsafe or premature proposals; and a frozen base motion diffusion transformer with per-skill ControlNet adapters realizes each skill as a half-second full-body motio

Load-bearing premise

The half-physics simulator removes balance and motor-tracking failures exactly, while preserving contact, collision, gravity, and object displacement without introducing motor artifacts—so every failure can be attributed to the decision maker.

What would settle it

Run the same 1,218 episodes under a fully dynamic torque-controlled humanoid and under pure kinematic playback. If half-physics outcomes diverge from both, or if commanded skill parameters are realized differently when the body is pressed against obstacles than in free space, the decoupling claim collapses. A simpler probe: check whether the passive stiffness λ=1.0 and 120 Hz integration yield jamming or sliding that a torque controller would not produce.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Embodied evaluation can now separate decision quality from motor skill, so VLM reasoning can be compared across embodiments without a locomotion confound.
  • The bottleneck being self-localization and termination suggests proprioceptive or contact feedback, or persistent spatial memory, may be more valuable than larger context windows.
  • The plug-and-play skill adapters mean new action vocabularies can be added without retraining the decision maker, so benchmark coverage can grow cheaply.
  • The staged metrics and root-cause labels turn the benchmark into a diagnostic: progress on body awareness should show up as higher navigation and interaction success, not just better target finding.
  • The verifier ablation (navigation success drops from 27.0% to 2.0% without it) shows that short-context verification, not longer memory, is what closes the loop in this setting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to feed the VLM a proprioceptive or contact signal; the paper's own discussion predicts this may be the missing input, and if success rises substantially, 'embodied self-awareness' would be a sensor-channel problem as much as a reasoning deficit.
  • The half-physics premise is unvalidated against a full-dynamics baseline; if kinematic driving distorts contact outcomes (jamming, sliding, unrealistic blocking), some attributed decision failures could actually be simulator artifacts. A direct comparison would settle this.
  • The same half-physics loop should generalize to manipulation and tool use, where the finest body-placement failures (sitting on air, wrong object) become hand and tool placement failures—an even stiffer test of anticipatory spatial reasoning.
  • Because the interface omits tactile feedback, the benchmark may underestimate models that could otherwise use contact information; a purely vision-based ceiling may not be the ceiling for the same reasoner with different sensors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces HumanCLAW, an evaluation framework intended to separate a VLM's action decision-making from low-level motor execution. A frozen off-the-shelf VLM issues atomic whole-body skill commands (walk, turn, sit, etc.) with continuous parameters; a skill-conditioned motion generator produces 0.5s kinematic motion chunks; and a 'half-physics' simulator (AI Habitat + Bullet, 120Hz, passive stiffness lambda=1.0) executes them under gravity and contact while abstracting away balance and torque-level motor control. The authors build HumanCLAW-Bench with 1,218 find-navigate-interact episodes across 41 HSSD indoor scenes, evaluate nine VLMs, and report that the best model completes only 16.8% of episodes. They argue that perception is not the bottleneck; rather, the models fail at embodied self-awareness: knowing where the body is, whether it has arrived, and whether it has collided. Success metrics are staged (FindSR/NavSR/InteractSR) with geometric objective criteria plus model-acknowledged completion, and root-cause labels are assigned by deterministic rules in Appendix B.

Significance. If the half-physics decoupling is sound, HumanCLAW provides a useful middle layer between symbolic agent benchmarks and full-dynamics VLA evaluation: it keeps a full body in a closed-loop physical world while removing balance and motor-tracking failures, and it ships a large benchmark with transparent, rule-based error attribution and per-stage metrics. The comparison across nine models and the skill-fidelity measurements are also useful empirical assets. However, the central scientific claim—that the reported failures 'belong to the decision maker' and reveal a missing embodied self-awareness—depends on a validation step that the paper does not provide: demonstrating that contact interactions in the half-physics simulator are free of motor-side artifacts. The paper's own limitation paragraph also concedes that the interface gives the agent no proprioceptive or contact signal, which confounds the 'missing faculty' interpretation. The contribution is valuable and the core idea is defensible, but the headline claims currently exceed what the evidence supports.

major comments (3)
  1. [Section 2.4 and Appendix B (Table 7)] The load-bearing premise is that half-physics 'factors out' motor execution while preserving physical interaction, and the abstract/Section 4.4 conclude 'with motor execution factored out by construction, these failures belong to the decision maker.' Table 1 supports free-space skill fidelity, but it does not characterize what happens when the velocity-driven body contacts a wall, stair edge, or movable object. The 'unaware jammed' rule (≥8 consecutive locomotion steps with displacement <0.1m) can be produced by a kinematic command pressing against rigid contact while the passive body fails to advance—an execution artifact rather than a decision to keep moving. Without a full-dynamics baseline, contact-penetration/foot-sliding metrics, and sensitivity analysis over λ=1.0 and Δt=1/120s across the 41 scenes, the decision-level attribution is unsupported.
  2. [Section 4.4 (Finding 6) and Section 6] The central conclusion that 'current VLMs lack embodied self-awareness' is confounded by the observation interface. The agent receives only egocentric RGB and text history, with no proprioceptive or contact signal, as Section 6 acknowledges: 'a body-state or contact signal could be the missing input rather than a missing faculty.' The benchmark therefore cannot distinguish a missing faculty from a missing input modality. The abstract and Finding 6 should be qualified accordingly; as written they overstate what the experiments establish.
  3. [Section 2.2, Table 3, and Section 6] The evaluated system is not the off-the-shelf VLM alone but the VLM plus the HumanCLAW harness, whose skill-specific verifier is an external component doing substantial decision-level work. Table 3 shows that removing the verifier collapses NavSR from 27.0% to 2.0% and InteractSR from 18.9% to 0.0%. The claim that failures 'belong to the decision maker' is therefore relative to a specific scaffold. The paper should state clearly that the benchmark evaluates VLMs under this harness, not the VLM's unaided action intelligence. This is acknowledged indirectly in Section 6 but not in the abstract.
minor comments (4)
  1. [Table 1 / Section 4.1] The text says the method 'tracks every commanded magnitude at ≈1,' but stair-climbing ratios are 0.794±0.019 and 0.738±0.052. These are stable gains, not near-unity tracking. The wording should distinguish displacement-tracking skills from skills with a calibratable gain.
  2. [Figure 8(b)] The listed percentages appear to sum to 102% (6+9+12+23+38+3+10+1). This is likely rounding, but the figure should be reconciled or a note added.
  3. [Section 2.4] The velocity conversion formula only states linear joint velocity; rotational components are described as 'computed from relative rotations in the corresponding tangent space.' The exact quaternion/log-map formulation should be given for reproducibility.
  4. [Table 3] The baseline on the 100-episode mini-val reports InteractSR=18.9% for Gemma-4-31B, while the full-val Table 2 reports 11.1% for the same model. The discrepancy is large enough that confidence intervals or a note on mini-val sampling variance should be provided.

Circularity Check

0 steps flagged

No significant circularity: benchmark results are measured, not derived; the half-physics dependency is an explicit design assumption (self-cited) rather than a circular reduction.

full rationale

The paper's central results are empirical measurements from a purpose-built benchmark: 1,218 episodes, geometric success criteria (≥100 semantic pixels, AABB distance <20 cm, pelvis mesh contact), and rule-based root-cause labels over logged rollouts. These criteria are not defined by the models' outputs, and the headline success rates (best 16.8% InteractSR) are observed, not predicted from the framework's assumptions. The load-bearing phrase 'with motor execution factored out by construction, these failures belong to the decision maker' is a conditional that depends on the half-physics simulator actually factoring out balance and motor tracking; the simulator is adopted from the authors' own prior work (Siyao et al., 2025) with λ=1.0 and 120 Hz. This is a self-citation and an unvalidated design premise, and the 'unaware jammed' rule is a behavioral proxy that could in principle be affected by contact artifacts. However, this is a correctness/validity risk, not circularity: the paper does not fit a parameter and then predict a closely related quantity, nor does it define the measured deficiencies in terms of the assumptions that are supposed to explain them. The limitations section explicitly acknowledges that half-physics abstracts away motor control and that decision-level attribution is relative to the skill vocabulary. Thus the empirical findings have independent content; the residual concern about half-physics fidelity does not make the derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The framework rests on hand-chosen evaluation thresholds, on the half-physics decoupling, on the curated skill-motion alignment, and on the interpretation of residual failures as a missing faculty rather than a sensory-deprivation artifact. No new physical entities are introduced; 'embodied self-awareness' is a construct summarizing measured failure patterns, which the authors themselves note could alternatively stem from the absence of a tactile or proprioceptive channel.

free parameters (5)
  • FindSR target visibility threshold = >=100 semantic pixels in 512x512 ego view
    Hand-chosen threshold defining target rendered; affects FindSR and the perception-versus-exploration attribution in Figure 8.
  • NavSR arrival distance = 20 cm to target AABB (1 m variant in Table 4)
    Hand-chosen distance for reached the target; the paper reports a 1 m variant, so the qualitative finding is partly robust, but exact percentages shift.
  • InteractSR success criterion = pelvis mesh contact with target at committed stop; only sit episodes
    Hand-chosen contact criterion for the single interaction modality; depends on the half-physics contact model.
  • Difficulty tier thresholds = distance 3.5/8 m; choice 2/5; obstacle 2.5/5
    Fixed thresholds splitting episodes into easy/medium/hard; used for stratified analysis, not for the headline success numbers.
  • Progressive subjective acknowledgment requirement = model must acknowledge completion in visible-state text and actively stop
    Requiring subjective acknowledgment is central to the self-awareness finding; it creates large gaps between GeoNavSR and NavSR and between GeoInteractSR and InteractSR.
axioms (5)
  • domain assumption The half-physics simulator (kinematic velocity driving with passive stiffness) removes balance and motor-tracking failures while preserving contact, collision, and gravity effects.
    Invoked in Section 2.4 and relied on for every decision-level attribution ('failures belong to the decision maker'). Not validated against a full-dynamics baseline in these scenes; if false, Nav/Interact failures could be motor artifacts.
  • domain assumption AMASS clips curated and filtered for each skill are single-purpose and commonsense-aligned, so a skill ControlNet realizes the ordinary meaning of the skill name.
    Section 2.3; Table 1 reports fidelity ratios, but semantic alignment (e.g., that walk means a normal walk in all contexts) is asserted, not measured on the benchmark scenes.
  • domain assumption A* on a 5 cm occupancy grid reconstructs navigable routes that match HSSD geodesic distances (median ratio 0.99).
    Section 3.1; used for difficulty stratification and route-based metrics; the 0.99 median ratio is reported, but residual errors on hard scenes are unknown.
  • domain assumption The harness prompt and skill verifier do not inject the measured capability (self-awareness); residual failures are attributable to the frozen VLM.
    Section 2.2 and 4.3. The verifier is hand-built by the authors and restores about 25 points of NavSR in the ablation; whether remaining failures reflect the model rather than the harness is an assumption central to the embodied self-awareness conclusion.
  • standard math Flow matching objective and the 30-step midpoint ODE solver correctly produce the motion chunks used in evaluation.
    Eq. (5) and inference with a midpoint solver; standard practice, not independently verified here.
invented entities (1)
  • Embodied self-awareness (as a postulated missing faculty) no independent evidence
    purpose: Explains the observed failure patterns (arrival misjudgment, jamming unawareness, sitting into thin air) across nine VLMs; proposed as the bottleneck distinguishing seeing from acting.
    It is a summary construct derived from the benchmark's own failure taxonomy, not an independently measured mechanism. The paper itself (Section 6) notes that a missing proprioceptive or tactile input channel could be the cause rather than a missing faculty, so the construct is not independently evidenced outside this benchmark.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HumanCLAW: Can Vision-Language Models Act Through a Body?." pith.science (2026). https://pith.science/paper/FTGAKXJR

@misc{pith2026260727180,
  author       = {Pith},
  title        = {Pith review of: HumanCLAW: Can Vision-Language Models Act Through a Body?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTGAKXJR}},
  note         = {Machine review of arXiv:2607.27180}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 37 canonical work pages · 1 internal anchor

  1. [1]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko S \"u nderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    work page 2018

  2. [2]

    _0 : A vision-language-action flow model for general robot control, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. _0 : A vis...

    Pith/arXiv arXiv 2024

  3. [3]

    Turner, Eric Undersander, and Tsung-Yen Yang

    Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. PARTNR : A benchmark for planning...

    work page 2025

  4. [4]

    SpatialVLM : Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM : Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455--14465, 2024

    work page 2024

  5. [5]

    LoTa-Bench : Benchmarking language-oriented task planners for embodied agents

    Jae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim, and Minsu Jang. LoTa-Bench : Benchmarking language-oriented task planners for embodied agents. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=ADSxCpCu9s

    work page 2024

  6. [6]

    Umo: Unified in-context learning unlocks motion foundation model priors

    Xiaoyan Cong, Zekun Li, Zhiyang Dou, Hongyu Li, Omid Taheri, Chuan Guo, Abhay Mittal, Sizhe An, Taku Komura, Wojciech Matusik, et al. Umo: Unified in-context learning unlocks motion foundation model priors. arXiv preprint arXiv:2603.15975, 2026

    work page arXiv 2026

  7. [7]

    Bullet physics simulation

    Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, page 7:1. ACM, 2015. doi:10.1145/2776880.2792704

    work page arXiv 2015

  8. [8]

    Moving by looking: Towards vision-driven avatar motion generation

    Markos Diomataris, Berat Mert Albaba, Giorgio Becherini, Partha Ghosh, Omid Taheri, and Michael J Black. Moving by looking: Towards vision-driven avatar motion generation. arXiv preprint arXiv:2509.19259, 2025

    work page arXiv 2025

  9. [9]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E : An embodied...

    work page 2023

  10. [10]

    Manipulate-anything: Automating real-world robots using vision-language models

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models. In Conference on Robot Learning, 2024

    work page 2024

  11. [11]

    Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine

    Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems...

    doi:10.15607/rss.2024.xx.090 2024

  12. [12]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152--5161, 2022. doi:10.1109/CVPR52688.2022.00509

    arXiv 2022

  13. [13]

    MoMask : Generative masked modeling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. MoMask : Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900--1910, 2024. https://openaccess.thecvf.com/content/CVPR2024/html/Guo_MoMask_Generative_Masked_Modeling_of_3D_Human_Motions_CVPR_2024_paper.html

    work page 1900

  14. [14]

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J. Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11354--11364, 2021. doi:10.1109/ICCV48922.2021.01118

    arXiv 2021

  15. [15]

    ESI-Bench : Towards embodied spatial intelligence that closes the perception-action loop

    Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, and Yejin Choi. ESI-Bench : Towards embodied spatial intelligence that closes the perception-action loop. arXiv preprint arXiv:2605.18746, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    VoxPoser : Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser : Composable 3d value maps for robotic manipulation with language models. In Proceedings of the 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 540--562. PMLR, 2023 a . https://proceedings.mlr.press/v229/huang23b.html

    work page 2023

  17. [17]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the 6th Conference on Rob...

    work page 2023

  18. [18]

    Como: Controllable motion generation through language guided pose code editing

    Yiming Huang, Weilin Wan, Yue Yang, Chris Callison-Burch, Mark Yatskar, and Lingjie Liu. Como: Controllable motion generation through language guided pose code editing. In European Conference on Computer Vision, pages 180--196. Springer, 2024

    work page 2024

  19. [19]

    Do as i can, not as i say: Grounding language in robotic affordances

    Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 287--318. PMLR, 2023...

    work page 2023

  20. [20]

    IAM: Identity-Aware Human Motion and Shape Joint Generation

    Wenqi Jia, Zekun Li, Abhay Mittal, Chengcheng Tang, Chuan Guo, Lezi Wang, James Matthew Rehg, Lingling Tao, and Size An. Iam: Identity-aware human motion and shape joint generation. arXiv preprint arXiv:2604.25164, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    Chang, and Manolis Savva

    Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat synthetic scenes dataset ( HSSD -200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    arXiv 2024

  22. [22]

    Foster, Pannag R

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P. Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA : An open-source vision-language-action model. In Proceedings of the 8th Conference ...

    work page 2025

  23. [23]

    MolmoAct : Action reasoning models that can reason in space

    Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Han, Mohammadreza Salehi, Jae Sung Hwang, et al. MolmoAct : Action reasoning models that can reason in space. arXiv preprint arXiv:2508.07917, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    BEHAVIOR-1K : A benchmark for embodied AI with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart \'i n-Mart \'i n, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. BEHAVIOR-1K : A benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Mac...

    work page 2023

  25. [25]

    Unfolding spatial cognition: Evaluating multimodal models on visual simulations

    Linjie Li, Mahtab Bigverdi, Jiawei Gu, Zixian Ma, Yinuo Yang, Ziang Li, Yejin Choi, and Ranjay Krishna. Unfolding spatial cognition: Evaluating multimodal models on visual simulations. arXiv preprint arXiv:2506.04633, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Embodied agent interface: Benchmarking LLMs for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017...

    doi:10.52202/079017-3188 2024

  27. [27]

    Llamo: Scaling pretrained language models for unified motion understanding and generation with continuous autoregressive tokens

    Zekun Li, Sizhe An, Chengcheng Tang, Chuan Guo, Ivan Shugurov, Linguang Zhang, Amy Zhao, Srinath Sridhar, Lingling Tao, and Abhay Mittal. Llamo: Scaling pretrained language models for unified motion understanding and generation with continuous autoregressive tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

    work page 2026

  28. [28]

    Genhsi: Controllable generation of human-scene interaction videos

    Zekun Li, Rui Zhou, Rahul Sajnani, Xiaoyan Cong, Daniel Ritchie, and Srinath Sridhar. Genhsi: Controllable generation of human-scene interaction videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 138--149, 2026 b

    work page 2026

  29. [29]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, 2023

    work page 2023

  30. [30]

    Large model empowered embodied AI : A survey on decision-making and embodied learning, 2025

    Wenlong Liang, Rui Zhou, Yang Ma, Bing Zhang, Songlin Li, Yijia Liao, and Ping Kuang. Large model empowered embodied AI : A survey on decision-making and embodied learning, 2025. https://arxiv.org/abs/2508.10399

    Pith/arXiv arXiv 2025

  31. [31]

    VisualAgentBench : Towards large multimodal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. VisualAgentBench : Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    A survey on vision-language-action models for embodied AI

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied AI . IEEE Transactions on Neural Networks and Learning Systems, 2026. doi:10.1109/TNNLS.2025.3650584

    arXiv 2026

  33. [33]

    GR00T N1 : An open foundation model for generalist humanoid robots, 2025

    NVIDIA , Johan Bjorck, Fernando Casta \ n eda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, et al. GR00T N1 : An open foundation model for generalist humanoid robots, 2025. https://arxiv.org/abs/2503.14734

    Pith/arXiv arXiv 2025

  34. [34]

    VirtualHome : Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome : Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494--8502, 2018. https://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR...

    work page 2018

  35. [35]

    Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimir Vondrus, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara ...

    work page 2024

  36. [36]

    Habitat: A platform for embodied AI research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied AI research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339--9347, 2019. doi:10.1109/ICCV.2019.00943

    arXiv 2019

  37. [37]

    ALFRED : A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED : A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10740--10749, 2020. https://openaccess.thecvf.com/content_CV...

    work page 2020

  38. [38]

    Bailando: 3 D dance generation by actor-critic GPT with choreographic memory

    Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3 D dance generation by actor-critic GPT with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    work page 2022

  39. [39]

    Bailando++: 3 D dance GPT with choreographic memory

    Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando++: 3 D dance GPT with choreographic memory. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45 0 (12): 0 14192--14207, 2023

    work page 2023

  40. [40]

    Duolando: Follower GPT with off-policy reinforcement learning for dance accompaniment

    Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower GPT with off-policy reinforcement learning for dance accompaniment. In International Conference on Learning Representations (ICLR), 2024

    work page 2024

  41. [41]

    Half-physics: Enabling kinematic 3d human model with physical interactions

    Li Siyao, Yao Feng, Omid Taheri, Chen Change Loy, and Michael J Black. Half-physics: Enabling kinematic 3d human model with physical interactions. arXiv preprint arXiv:2507.23778, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner : Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998--3009, 2023. https://openaccess.thecvf.com/content/ICCV2023/html/Song_LLM-Planner_Few-Shot_Gr...

    work page 2023

  43. [43]

    Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra

    Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel X. Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Trainin...

    work page 2021

  44. [44]

    Cradle: Empowering foundation agents towards general computer control

    Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023. https://openreview.net/forum?id=SJ1kSyO2jwu

    work page 2023

  46. [46]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. https://openreview.net/forum?id=ehfRiF0R3a

    work page 2024

  47. [47]

    Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In Advances in Neural Information Processing Systems, volume 36, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/6b8dfb8c0c12e6fafc6c256cb08a5ca7-Abstract-...

    work page 2023

  48. [48]

    Text2interact: High-fidelity and diverse text-to-two-person interaction generation

    Qingxuan Wu, Zhiyang Dou, Chuan Guo, Yiming Huang, Qiao Feng, Bing Zhou, Jian Wang, and Lingjie Liu. Text2interact: High-fidelity and diverse text-to-two-person interaction generation. arXiv preprint arXiv:2510.06504, 2025

    arXiv 2025

  49. [49]

    OmniControl : Control any joint at any time for human motion generation

    Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. OmniControl : Control any joint at any time for human motion generation. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=gd0lAEtWso

    work page 2024

  50. [50]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    EmbodiedBench : Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench : Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. In Proceedings of the 42nd International Conference on Machine Lear...

    work page 2025

  52. [52]

    PhysDiff : Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. PhysDiff : Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16010--16021, 2023. doi:10.1109/ICCV51070.2023.01467

    arXiv 2023

  53. [53]

    Zhang, Thomas L

    Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench : Can vision-language models complete popular video games? arXiv preprint arXiv:2505.18134, 2025 a

    Pith/arXiv arXiv 2025

  54. [54]

    Egoreact: Egocentric video-driven 3d human reaction generation

    Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Rui Xu, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, et al. Egoreact: Egocentric video-driven 3d human reaction generation. arXiv preprint arXiv:2512.22808, 2025 b

    arXiv 2025

  55. [55]

    The wanderings of odysseus in 3d scenes

    Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20481--20491, 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Zhang_The_Wanderings_of_Odysseus_in_3D_Scenes_CVPR_2022_paper.html

    work page 2022

  56. [56]

    Yan Zhang, Yao Feng, Alp \'a r Cseke, Nitin Saini, Nathan Bajandas, Nicolas Heron, and Michael J. Black. PRIMAL : Physically reactive and interactive motor model for avatar learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12725--12736, 2025 c . https://openaccess.thecvf.com/content/ICCV2025/html/Zhang_PRIMAL_Phys...

    work page 2025

  57. [57]

    Synthesizing diverse human motions in 3d indoor scenes

    Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738--14749, 2023. https://openaccess.thecvf.com/content/ICCV2023/html/Zhao_Synthesizing_Diverse_Human_Motions_in_3D_Indoor_Scenes_ICCV_2023_paper.html

    work page 2023

  58. [58]

    RT-2 : Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. RT-2 : Vision-language-action models transfer web knowledge to robotic control. In Proceedings of the 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165--2183. PMLR, 2023. http...

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.