pith. machine review for the scientific record. sign in

arxiv: 2604.07430 · v1 · submitted 2026-04-08 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents

Tencent Robotics X , HY Vision Team: Xumin Yu , Zuyan Liu , Ziyi Wang , He Zhang , Yongming Rao , Fangfu Liu , Yani Zhang
show 13 more authors
Ruowen Zhao Oran Wang Yves Liang Haitao Lin Minghui Wang Yubo Dong Kevin Cheng Bolin Ni Rui Huang Han Hu Zhengyou Zhang Linus Shunyu Yao
Authors on Pith no claims yet

Pith reviewed 2026-05-10 18:29 UTC · model grok-4.3

classification 💻 cs.CV
keywords embodied foundation modelsmixture of transformersvision-language modelsvision-language-actionrobot controlspatial reasoningembodied reasoningself-evolving training
0
0 comments X

The pith

HY-Embodied-0.5 models use Mixture-of-Transformers and self-evolving training to boost perception and reasoning for embodied agents.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a family of vision-language foundation models built specifically for real-world robots and agents. It targets gaps in standard models by strengthening spatial and temporal visual perception plus embodied reasoning for prediction, interaction, and planning. The work introduces two sizes: a compact 2B-parameter version for edge use and a 32B version for harder tasks, both relying on a Mixture-of-Transformers design with latent tokens and an iterative self-evolving post-training process, plus distillation to move skills from large to small. Tests across 22 benchmarks show the smaller model beating comparable systems on 16 of them and the larger one reaching levels close to top models such as Gemini 3.0 Pro. The same base then supports training a Vision-Language-Action model that delivers solid results when controlling physical robots.

Core claim

The authors claim that a Mixture-of-Transformers architecture with latent tokens for modality-specific perception, paired with iterative self-evolving post-training and on-policy distillation, produces foundation models whose spatial-temporal perception and embodied reasoning exceed those of similarly sized general vision-language models, yielding benchmark gains and effective downstream robot control.

What carries the argument

Mixture-of-Transformers (MoT) architecture with latent tokens, which performs modality-specific computing to strengthen perceptual representations, together with the iterative self-evolving post-training paradigm that refines reasoning through repeated improvement cycles.

If this is right

  • The 2B model becomes practical for on-device robot deployment while retaining most capabilities.
  • Distillation transfers high-level reasoning from the 32B model to the smaller one without major loss.
  • The VLM foundation directly supports training VLA models that succeed in real physical robot evaluations.
  • Enhanced spatial and temporal perception enables better prediction and planning in dynamic environments.
  • Performance parity with frontier models at 32B scale suggests the method scales to complex embodied tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The design may reduce reliance on ever-larger datasets by focusing architectural and training innovations on perception and reasoning gaps.
  • Testing the models in long-horizon tasks with unpredictable physical feedback could reveal whether the self-evolving loop holds up beyond benchmark settings.
  • The same MoT-plus-distillation pattern might apply to other agent domains such as autonomous vehicles or household robots.
  • If the benchmarks under-represent certain failure modes, real-world robustness could still require additional safety layers not addressed here.

Load-bearing premise

The observed gains come mainly from the MoT design and self-evolving training rather than from training data scale, selection, or other unmentioned factors, and the 22 benchmarks fully represent the skills needed for real embodied agents.

What would settle it

A new physical robot task outside the 22 benchmarks where the resulting VLA model performs no better than a standard VLM baseline of similar size would indicate the approach does not deliver the claimed embodied advantages.

Figures

Figures reproduced from arXiv: 2604.07430 by Bolin Ni, Fangfu Liu, Haitao Lin, Han Hu, He Zhang, HY Vision Team: Xumin Yu, Kevin Cheng, Linus, Minghui Wang, Oran Wang, Rui Huang, Ruowen Zhao, Shunyu Yao, Tencent Robotics X, Yani Zhang, Yongming Rao, Yubo Dong, Yves Liang, Zhengyou Zhang, Ziyi Wang, Zuyan Liu.

Figure 1
Figure 1. Figure 1: Performance of HY-Embodied-0.5 MoT-2B on spatial and embodied benchmarks as well as downstream robot control tasks. HY-Embodied-0.5 pushes the frontier of embodied VLMs, while excelling in downstream real-world robot evaluations. 1 arXiv:2604.07430v1 [cs.CV] 8 Apr 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: HY-Embodied-0.5 Mixture-of-Transformers Architecture. The MoT design decouples the processing of visual and textual tokens by employing modality-specific QKV and FFN layers, alongside distinct attention mechanisms. Visual latent tokens and mixed optimization loss are employed to bridge and stress the relationships between modalities during large￾scale training. is an upgraded version of HY-ViT. Building up… view at source ↗
Figure 3
Figure 3. Figure 3: Attention Computation of our Modality-Adaptive MoT. We vi￾sualize the attention computation un￾der actual interleaved multi-modal sequences with distinct colors. Beyond employing the MoT architecture, we make further improvements to better model visual inputs. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Data Distribution for Pre-training and Mid-training Stages. We conduct large-scale embodied pre-training and mid-training to establish foundational and advanced physical￾world competencies. The pre-training mixture encompasses over 200B tokens based on spatial, robotics, and visual perception tasks. The mid-training stage leverages over 12M high-quality QA pairs for complex real-world execution based on di… view at source ↗
Figure 5
Figure 5. Figure 5: Training Pipeline for HY-Embodied-0.5 Series. Large-scale pre-training establishes the models’ foundational multi-modal representations and robust spatial-embodied percep￾tion. The subsequent Embodied Post-training phase explicitly enhances complex reasoning capabilities through iterative self-evolution and reinforcement learning. Finally, we employ on-policy distillation to effectively transfer the knowle… view at source ↗
Figure 6
Figure 6. Figure 6: Reward Designs for Embodied Reinforcement Learning. To accommodate diverse embodied tasks during RL optimization, we systematically formulate reward functions into four categories: Grounding-Based for spatial localization, Regression-Based for numerical estimation, Trajectory-Based for motion and planning, and Textual-Based for general and semantic reasoning. estimation, and open-ended reasoning. A single … view at source ↗
Figure 7
Figure 7. Figure 7: Performance on General Understanding Benchmark. Comparison of HY-Embodied￾0.5 MoT-2B with size-matched general VLMs. The results demonstrate that while our model is specifically optimized for spatial and embodied reasoning, it successfully maintains com￾parable and highly competitive performance across diverse general visual understanding tasks. well as document parsing and text-centric visual question ans… view at source ↗
Figure 8
Figure 8. Figure 8: Visualization Results on Visual Perception Tasks. Empowered by our comprehen￾sive visual perception training, HY-Embodied-0.5 MoT-2B demonstrates superior proficiency across foundational vision tasks, including depth estimation, object detection, and complex counting, outperforming competing embodied-specific and general VLMs. Embodied￾Task retrieve an onion from the shelf, place the onion into the plastic… view at source ↗
Figure 9
Figure 9. Figure 9: Visualization Results on Embodied Tasks. Our model demonstrates comprehensive proficiency across embodied tasks, including precise visual grounding, logical action plan￾ning, and scene understanding. scenarios, the model showcases strong sequential reasoning. Given a high-level objective and a history of completed steps, it accurately deduces the logical next actions—whether it involves determining the seq… view at source ↗
Figure 10
Figure 10. Figure 10: Illustration of Chain-of-Thought Process. Our HY-Embodied series demonstrates exceptional long-chain reasoning capabilities when tackling complex visual and embodied challenges. Regarding the specific thought process, rather than simply guessing outcomes, the models systematically analyze spatial relationships and affordances step-by-step, exhibit￾ing advanced self-reflection and error-correction mechanis… view at source ↗
Figure 11
Figure 11. Figure 11: MoT architecture enables faster convergence than the standard transformers (left), while delivering comparable inference speed (right). (a) presents the training loss curves, and (b) details the inference efficiency by comparing the total inference time, theoretical total FLOPs, prefill time, and decode time across different models. Input Image Attention on Visual Tokens Attention on Language Tokens Input… view at source ↗
Figure 12
Figure 12. Figure 12: Attention Visualizations for Visual Latent Tokens. Visual attention accurately localizes salient objects and key spatial regions, while language attention concurrently focuses on the corresponding core semantic entities, states, and action instructions. Efficiency for HY-Embodied-0.5 MoT. The proposed Mixture-of-Tokens (MoT) architecture demonstrates highly desirable characteristics, achieving faster conv… view at source ↗
Figure 13
Figure 13. Figure 13: Robot Experimental Setup and Success Rates for the Evaluated Tasks. (a) Real￾world setup of three representative tasks. Our platform employs a dual-arm Xtrainer equipped with head-mounted and wrist-mounted cameras. The benchmark includes: (1) Precision Plug-in Packing, (2) Tableware Stacking, and (3) Mug Hanging. Success rates are evaluated over 20 trials per task. (b) Object poses are randomly initialize… view at source ↗
read the original abstract

We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents. To bridge the gap between general Vision-Language Models (VLMs) and the demands of embodied agents, our models are developed to enhance the core capabilities required by embodied intelligence: spatial and temporal visual perception, alongside advanced embodied reasoning for prediction, interaction, and planning. The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning. To support the fine-grained visual perception essential for embodied tasks, we adopt a Mixture-of-Transformers (MoT) architecture to enable modality-specific computing. By incorporating latent tokens, this design effectively enhances the perceptual representation of the models. To improve reasoning capabilities, we introduce an iterative, self-evolving post-training paradigm. Furthermore, we employ on-policy distillation to transfer the advanced capabilities of the large model to the smaller variant, thereby maximizing the performance potential of the compact model. Extensive evaluations across 22 benchmarks, spanning visual perception, spatial reasoning, and embodied understanding, demonstrate the effectiveness of our approach. Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro. In downstream robot control experiments, we leverage our robust VLM foundation to train an effective Vision-Language-Action (VLA) model, achieving compelling results in real-world physical evaluations. Code and models are open-sourced at https://github.com/Tencent-Hunyuan/HY-Embodied.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces HY-Embodied-0.5, a family of embodied foundation models (2B and 32B activated-parameter variants) built on a Mixture-of-Transformers (MoT) architecture with latent tokens for spatial-temporal perception and an iterative self-evolving post-training paradigm for embodied reasoning. It reports that the 2B model outperforms similarly sized SOTA models on 16 of 22 benchmarks spanning visual perception, spatial reasoning, and embodied understanding, that the 32B variant reaches performance comparable to Gemini 3.0 Pro, and that the models enable effective Vision-Language-Action (VLA) policies with compelling real-world robot results; code and models are open-sourced.

Significance. If the benchmark gains and downstream robot results prove robust, the work would advance embodied AI by supplying purpose-built, deployable VLMs that explicitly target the perception-reasoning loop required for physical agents. The open-sourcing of both model weights and training code is a concrete strength that would allow independent verification and extension.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Experimental Evaluation): The central claims of outperformance on 16 benchmarks and 'compelling results in real-world physical evaluations' are asserted without any tabulated per-benchmark scores, baseline descriptions, error bars, statistical tests, or ablation studies isolating the contribution of the MoT architecture versus data scale or curation; this absence directly prevents verification of the performance claims that constitute the paper's primary contribution.
  2. [§3.2 and §4] §3.2 (Self-evolving post-training) and §4: No controlled comparison is presented that isolates the effect of the iterative self-evolving paradigm from the underlying data mixture or from standard supervised fine-tuning; without such an ablation the attribution of gains to the proposed training method remains unsupported and load-bearing for the architectural novelty claim.
minor comments (2)
  1. [Abstract] The phrase 'activated parameters' is used without definition; a brief clarification of how this differs from total parameter count in the MoT design would improve readability for readers unfamiliar with sparse activation.
  2. [§4] The 22 benchmarks are listed only by category; an explicit table or appendix enumerating each benchmark name, metric, and source would make the evaluation section self-contained.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. The comments have helped us strengthen the rigor and verifiability of our experimental claims. We have revised the manuscript to include consolidated tables, error bars, statistical tests, and targeted ablations as requested. Below we respond point by point to the major comments.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experimental Evaluation): The central claims of outperformance on 16 benchmarks and 'compelling results in real-world physical evaluations' are asserted without any tabulated per-benchmark scores, baseline descriptions, error bars, statistical tests, or ablation studies isolating the contribution of the MoT architecture versus data scale or curation; this absence directly prevents verification of the performance claims that constitute the paper's primary contribution.

    Authors: We acknowledge that a single, consolidated presentation of all per-benchmark results was not sufficiently prominent. In the revised manuscript we have added Table 1 in §4, which reports exact scores for every one of the 22 benchmarks for both the 2B and 32B models together with the corresponding baselines, standard deviations, and p-values from paired statistical tests. We have also expanded the real-world robot section with quantitative success rates and setup details. In addition, we include a new ablation that directly compares the MoT architecture against a standard transformer backbone trained on identical data scale and curation, thereby isolating the architectural contribution. revision: yes

  2. Referee: [§3.2 and §4] §3.2 (Self-evolving post-training) and §4: No controlled comparison is presented that isolates the effect of the iterative self-evolving paradigm from the underlying data mixture or from standard supervised fine-tuning; without such an ablation the attribution of gains to the proposed training method remains unsupported and load-bearing for the architectural novelty claim.

    Authors: We agree that an explicit controlled ablation is required to substantiate the contribution of the self-evolving paradigm. We have added a new ablation study in §4 that trains otherwise identical models with and without the iterative self-evolving post-training stage, using the exact same data mixture and comparing against standard supervised fine-tuning. The results demonstrate consistent additional gains attributable to the self-evolving procedure. Section 3.2 has been updated to reference this ablation. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents architectural and training design choices (MoT with latent tokens, iterative self-evolving post-training, on-policy distillation) as explicit engineering decisions to address embodied agent needs, followed by empirical reporting of benchmark results and downstream robot evaluations against external references. No equations, parameter-fitting steps presented as predictions, self-definitional reductions, or load-bearing self-citations appear in the text. Claims rest on independent benchmark comparisons and open-sourced code/models, rendering the derivation chain self-contained without any step reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract describes architectural and training innovations but introduces no explicit free parameters, mathematical axioms, or new invented entities beyond standard components of vision-language models.

pith-pipeline@v0.9.0 · 5684 in / 1241 out tokens · 71323 ms · 2026-05-10T18:29:06.287077+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

    cs.RO 2026-04 unverdicted novelty 5.0

    STARRY uses unified diffusion to align spatial-temporal world predictions with action generation plus GASAM for geometry-aware attention, reaching 93.82%/93.30% success on 50 bimanual tasks in simulation and raising r...

Reference graph

Works this paper leans on

6 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    DeepSeek-V3 Technical Report

    URLhttps://arxiv.org/abs/2412.19437. DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, P . Wang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.Nature, 645:633–638, 2025. https: //doi.org/10.1038/s41586-025-09422-z. M. Dehghani, B. Mustafa, J. Djolonga, J. Heek, M. Minderer, M. Caron, A. Steiner, J. Puigcerver, R....

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1038/s41586-025-09422-z 2025

  2. [2]

    move the spatula towards the front edge of the table

    [gripper], then (490, 340) [midway to towel], then (634, 518) [midway to target], then (701, 614) [target lower right]. [ ... ] Yeah, that makes sense. Response:[<point>(417,279)</point>,<point>(528,387)</point>,<point>(657,553)</point>] Trajectory Case 1 Fig.B1:Example of end-effector trajectory prediction. 31 Tencent HY Trajectory Prediction under Joint...

  3. [3]

    第4张：首先处理花材 ，修剪花的花茎和叶片 ，为插花做准备 。

  4. [4]

    第2张：选取修剪好的花材 ，准备进行插花布置 。

  5. [5]

    第3张：将选好的花材插入花篮的基底中 ，进行插花操作 。

  6. [6]

    Because the floor might be slippery

    第1张：花篮制作完成 ，成品呈现。 Reasoning Task – Sorting Case Fig.B14:Example of sorting-based reasoning task. 44 Tencent HY Counting-based Reasoning Task. Question: These are frames of a video. How many trash bin(s) are in this room? Please answer the question using a single word or phrase. Thinking Process: Got it, let's see. The trash cans: there's a blue one and a ...