REVIEW 5 major objections 5 minor 2 cited by
CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion
T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A humanoid locomotion policy can traverse stairs, gaps, and platforms by learning directly from raw depth and body state, with no terrain map and no real-world fine-tuning.
desk verdict A solid single-stage depth-conditioned humanoid locomotion paper with good ablations and credible hardware trials, but the zero-shot transfer claim is weaker than the prose suggests because clean simulated depth is never stress-tested against real sensor noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the interaction between the proprioception token and depth tokens: the proprioceptive state acts as the query in multi-head cross-modal attention over local depth tokens, so the visual feature the policy sees is selected by what the body is currently doing. Gated residual fusion then combines this selected depth feature with proprioception through a learned channel-wise gate while keeping a residual path; the recurrent fusion module integrates the result over time with a Gated Recurrent Unit (GRU) and a highway output gate that lets the policy depend more on memory during flight phases and risky postures. The foothold-placement reward is the other key object: it ext
What would settle it
A controlled hardware experiment: run the same traversal protocol on a 15 cm/30 cm staircase under (a) normal indoor light, (b) strong oblique sunlight, and (c) a reflective panel placed beside the riser; then compare success rates. Because training used no synthetic depth corruption, a large drop in (b) or (c) would directly contradict the claim that the clean-rendered depth distribution transfers zero-shot.
Extended reading notes
Core claim
The central claim is that a humanoid policy trained end-to-end on raw depth and proprioception can traverse geometry-heavy terrain at leading difficulty levels without any explicit geometric intermediate or auxiliary geometry supervision. To support this, the paper introduces CReF's three fusion mechanisms—proprioception-queried cross-modal attention, gated residual fusion, and recurrent fusion with a highway output gate—and shows through ablations that each contributes, with cross-modal attention being the most critical. The terrain-aware foothold placement reward, which extracts supportable planar footholds from foot-centric point-cloud windows and rewards touchdown near them, tightens foo
Load-bearing premise
The zero-shot transfer claim rests on the assumption—explicit in the paper's training setup and acknowledged in its limitations—that depth rendered from clean first-hit ray-capsule intersections is representative enough of the real camera's output that a policy never exposed to synthetic depth corruption stays reliable when faced with reflective holes, handrails, and cluttered scenes.
Editorial extensions
If this is right
- Perceptive humanoid locomotion can be trained in a single stage: no elevation map construction, no multi-stage skill transfer, and no geometry-shaped auxiliary targets.
- Stair descent, the hardest failure regime, is improved most by the foothold-placement reward, which concentrates touchdown distributions and removes ascent ankle-riser collisions.
- Removing cross-modal attention is the most damaging ablation, so state-conditioned depth selection is the load-bearing architectural choice.
- The highway gate behaves as designed: it activates more on step-like terrain, during flight phases, and in risky postures, consistent with state-dependent memory use.
- The method generalizes beyond training difficulty in simulation and to some out-of-distribution real scenes, suggesting the learned representation preserves terrain-relevant cues without explicit geometry.
Reading between the lines
- If the central claim holds, the bottleneck for perceptive humanoid locomotion shifts from representation design to sensor fidelity and latency: policies could be simpler, but depth cameras with reflective holes or missing pixels would need to be handled at the sensor or training level.
- A testable extension is to inject realistic depth corruption (specular holes, invalid pixels, noise) during training; the paper explicitly trains without it, so a comparison against the clean-depth version would quantify how much of the real-world robustness comes from the architecture versus the sensor's relative cleanliness in the tested scenes.
- The foothold-candidate extraction, based on local planarity of point-cloud windows, is platform-agnostic; it could be applied to other legged robots or combined with footstep planning, not just used as a reward signal.
- Because the policy was trained with a single fixed depth latency, the framework's robustness to latency jitter is untested; an evaluation with randomized frame delays would show whether the recurrent fusion genuinely covers temporal uncertainty or just matched the fixed pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CReF, a single-stage depth-conditioned humanoid locomotion policy that maps raw depth and proprioception directly to joint position targets, without an explicit 2.5D terrain map or geometric reconstruction. The architecture combines a depth tokenizer, proprioception-queried cross-modal attention, gated residual fusion, and a GRU with a highway output gate. A terrain-aware foothold placement reward extracts supportable foothold candidates from foot-end point clouds and shapes touchdown locations. The authors evaluate CReF in Isaac Gym across stair, gap, and platform terrains, compare against architectural ablations and the HPL baseline, analyze the foothold distribution and highway-gate behavior, and report zero-shot hardware trials on an AGIBOT X2 Ultra, including stair, platform, gap, and indoor OOD tasks.
Significance. If fully established, CReF would provide evidence that a relatively simple single-stage depth-plus-proprioception policy can match or exceed geometry-mediated perceptive baselines on structured terrain, and that a geometric foothold reward can improve contact repeatability. The paper's strengths include a systematic architecture ablation, matched training protocols, a foothold-reward comparison, highway-gate behavior analysis, and a nontrivial hardware deployment with a large number of indoor trials. The significance is currently bounded by the absence of quantitative evidence on robustness to real depth-sensor artifacts, by missing comparisons to the closest raw-depth baselines, and by the lack of variance estimates for both simulation and hardware results.
major comments (5)
- [§II-D, §III-C, Fig. 5] The zero-shot transfer claim in Contribution 3 ('remains effective in challenging real-world scenes') is not quantified. Section II-D explicitly states that no synthetic depth corruption is injected during training, while Fig. 5 highlights real scenes with severe reflection and invalid depth holes. Table IV reports only standardized indoor tasks; no success metric is given for the degraded outdoor/reflective scenes. Since the policy's depth encoder is trained on clean rendered depth, it is unknown whether it operates inside its training distribution in these cases. Add quantitative real trials in degraded scenes and/or a simulated sensor-noise and depth-dropout evaluation to substantiate the robustness claim.
- [Table IV] The hardware success rates (20/20, 20/20, 18/20, 19/20) are single-run point estimates with no confidence intervals or cross-trial variance. The trial protocol is not defined: how terrain properties vary across trials, whether failures are reset, or what exactly constitutes success. For 18/20, the 95% CI is roughly [0.69, 0.99]; for 19/20, [0.76, 1.0]. Report more trials, confidence intervals, or a detailed protocol so the hardware claim is not dependent on a single run.
- [Table II] The simulation ablations are presented without multiple seeds or standard deviations. The 'consistently outperforms' claim is based on 2000-environment rollouts from a single training run. On easy settings, differences are small (e.g., Full CReF 99.85 vs. w/o Cross-Attn 96.25 at Easy stairs), and without seed variance one cannot assess significance. Report mean and standard deviation over at least three training seeds for the main comparisons.
- [§III-A and references [24],[25]] The most closely related single-stage depth-conditioned methods, Hiking in the Wild [24] and Now You See That [25], are cited in the introduction but not included in the comparison. Since the paper's contribution is a raw-depth policy, a comparison against these raw-depth baselines, or a reasoned exclusion, is needed to support the claim that CReF improves over the prior single-stage depth-conditioned state of the art. The HPL baseline alone is geometry-mediated and does not cover this class.
- [§III-B2 and Table I] Contribution 2 claims the foothold placement reward yields 'substantial gains in long-duration stair traversal, especially in descent,' but the paper reports only touchdown distributions for the FCQR comparison (Fig. 4), not success rates or failure counts. Because the reward directly maximizes proximity to supportable candidate windows, the tighter distribution in Fig. 4 is partly built into the objective; the traversal-level benefit must be demonstrated with ascent/descent success rates for the reward ablation.
minor comments (5)
- [§I] Duplicate phrase: 'directly maps onboard proprioception and forward-facing depth directly to joint position targets' — remove one 'directly'.
- [Table III] The Full CReF row reports only seven failure counts while other rows report eight. The OOD columns should be labeled clearly and the ascent/descent counts completed or explained.
- [Eq. (20)] The thresholds r_th, eta_th, h_min, and the tolerance s_xz are not given values. Report the chosen values and any sensitivity analysis, since the foothold reward's behavior depends critically on them.
- [Abstract and §I] The phrasing 'without explicit geometric intermediates or auxiliary geometry-related targets' should be qualified: the foothold reward computes planarity, normal alignment, and recessed-height criteria from point clouds, which is a geometric shaping signal, even though it is not a map or reconstruction.
- [Fig. 3] Provide more detail on the MuJoCo OOD terrains: their geometric composition, how they are generated, and why they are outside the training distribution.
Circularity Check
No significant circularity: the central claims are empirical benchmarking results, and the only self-referential element (foothold reward shaping the measured touchdown distribution) is a disclosed training objective, not a hidden prediction.
full rationale
I walked the paper's derivation chain. The central contribution—depth-conditioned humanoid locomotion via cross-modal attention, gated residual fusion, and recurrent fusion—is an architecture trained with PPO and evaluated by terrain success rates against ablations and a re-implemented baseline (Tables II–III, Section III-B1). These are empirical comparisons, not analytic derivations from the method definition, so no claim reduces to its inputs by construction. The terrain-aware foothold placement reward (Eq. 23) does by definition reward touchdown positions near supportable candidates, and the paper explicitly discloses that the resulting tighter touchdown distribution is 'consistent with its design' (Section III-B2). This is a reward-ablation result, not a fitted parameter renamed as a prediction; the empirical question is whether the reward improves traversal, which is tested via FCQR comparison. The zero-shot hardware transfer claim (Section III-C) is an empirical outcome of a training choice—'No synthetic depth corruption is injected during training' (Section II-D)—and the paper's own limitation statement acknowledges sensitivity to illumination and reflective surfaces, making that a correctness risk rather than a circularity. Self-citations ([19], [20], [22]) appear only in related-work framing and are not load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. The results are self-contained against ablations and task benchmarks; the only minor self-referential aspect is the reward-design/foothold-metric alignment, which is disclosed and not a hidden identity. Score 1 reflects this minor, non-load-bearing self-consistency rather than any circular derivation.
Assumptions & free parameters
free parameters (8)
- roughness threshold r_th
- normal-alignment threshold eta_th
- minimum foothold height h_min
- foothold distance tolerance s_xz
- candidate window size and stride =
24 cm x 10 cm, stride 4 cm
- reward weights in Table I =
e.g., foothold placement 2.0, base height 0.8, orientation 1.0
- model widths (token dimension d, token count N, MLP/GRU sizes)
- PPO hyperparameters and terrain randomization schedule
assumptions (5)
- domain assumption Clean simulated depth with no synthetic corruption is representative enough for zero-shot sim-to-real transfer on the D435i.
- ad hoc to paper A foothold is supportable iff the local point-cloud window is sufficiently planar, approximately horizontal, and not recessed (Eq. 20).
- domain assumption Reward weights and low-speed-aware terms from prior humanoid reward recipes [3,27] transfer to AGIBOT X2 Ultra.
- domain assumption PPO with 4096 parallel environments converges to a policy representative of the method.
- standard math Standard eigen-decomposition, backpropagation, and PPO are taken as correct.
Cite this review
Pith. "Pith review of CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion." pith.science (2026). https://pith.science/paper/C3JM7Z2K
@misc{pith2026260329452,
author = {Pith},
title = {Pith review of: CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion},
year = {2026},
howpublished = {\url{https://pith.science/paper/C3JM7Z2K}},
note = {Machine review of arXiv:2603.29452}
}
read the original abstract
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. While effective, these approaches introduce additional map-construction procedures or multi-stage skill-transfer processes beyond direct depth-to-control learning. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 2 Pith papers
-
Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation
A single neural-network policy, trained in simulation, makes a humanoid climb, vault, and traverse uneven terrain from onboard depth and a velocity command, with no skill labels or runtime motion graphs.
-
PAC-MAN: Perception-Aware CBF-RL for Whole-Body Safety in Humanoid Dodgeball
Link-CBF training with masked onboard depth lets a humanoid dodge whole-body contact near oracle rates in sim and on 95% of real throws, while stronger Joint-CBF needs better ball observability.
Reference graph
Works this paper leans on
-
[24]
Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids,
S. Zhu, Z. Zhuang, M. Zhao, K.-Y . Lee, and H. Zhao, “Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids,” arXiv preprint arXiv:2601.07718, 2026
arXiv 2026
-
[25]
Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels,
W. Sun, Y . Su, L. Huang, A. Zhang, D. Wei, M. San, D. Tian, E. Cao, F. Yan, E. Xie, and Z. Xie, “Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels,”arXiv preprint arXiv:2602.06382, 2026
arXiv 2026
-
[1]
Learning agile and dynamic motor skills for legged robots,
J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V . Tsounis, V . Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,”Science Robotics, vol. 4, no. 26, p. eaau5872, 2019
2019
-
[2]
Learning quadrupedal locomotion on deformable terrain,
S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo, “Learning quadrupedal locomotion on deformable terrain,”Science Robotics, vol. 8, no. 74, p. eade2256, 2023
2023
-
[3]
Revisiting reward design and evaluation for robust humanoid standing and walking,
B. van Marum, A. Shrestha, H. Duan, P. Dugar, J. Dao, and A. Fern, “Revisiting reward design and evaluation for robust humanoid standing and walking,” in2024 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS). IEEE, 2024, pp. 11 256–11 263
2024
-
[4]
Learning agile loco- motion on risky terrains,
C. Zhang, N. Rudin, D. Hoeller, and M. Hutter, “Learning agile loco- motion on risky terrains,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 11 864– 11 871
2024
-
[5]
Learning Humanoid Locomotion with Perceptive Internal Model,
J. Long, J. Ren, M. Shi, Z. Wang, T. Huang, P. Luo, and J. Pang, “Learning Humanoid Locomotion with Perceptive Internal Model,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 9997–10 003
2025
-
[6]
Gait-Adaptive Perceptive Humanoid Locomotion with Real-Time Under-Base Terrain Reconstruction,
H. Song, H. Zhu, T. Yu, Y . Liu, M. Yuan, W. Zhou, H. Chen, and H. Li, “Gait-Adaptive Perceptive Humanoid Locomotion with Real-Time Under-Base Terrain Reconstruction,”arXiv preprint arXiv:2512.07464, 2025
arXiv 2025
Show all 31 references
-
[7]
Perceptive locomotion in rough terrain–online foothold optimization,
F. Jenelten, T. Miki, A. E. Vijayan, M. Bjelonic, and M. Hutter, “Perceptive locomotion in rough terrain–online foothold optimization,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5370–5376, 2020
2020
-
[8]
Perceptive locomotion through nonlinear model-predictive control,
R. Grandia, F. Jenelten, S. Yang, F. Farshidian, and M. Hutter, “Perceptive locomotion through nonlinear model-predictive control,” IEEE Transactions on Robotics, vol. 39, no. 5, pp. 3402–3421, 2023
2023
-
[9]
Learning robust perceptive locomotion for quadrupedal robots in the wild,
T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,”Science Robotics, vol. 7, no. 62, p. eabk2822, 2022
2022
-
[10]
BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds,
H. Wang, Z. Wang, J. Ren, Q. Ben, T. Huang, W. Zhang, and J. Pang, “BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds,” inRobotics: Science and Systems (RSS), 2025
2025
-
[11]
Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds,
M. Dai, W. D. Compton, J. Li, L. Yang, and A. D. Ames, “Walk the PLANC: Physics-Guided RL for Agile Humanoid Locomotion on Constrained Footholds,”arXiv preprint arXiv:2601.06286, 2026
2026
-
[12]
DTC: Deep tracking control,
F. Jenelten, J. He, F. Farshidian, and M. Hutter, “DTC: Deep tracking control,”Science Robotics, vol. 9, no. 86, p. eadh5401, 2024
2024
-
[13]
Attention-based map encoding for learning generalized legged locomotion,
J. He, C. Zhang, F. Jenelten, R. Grandia, M. B ¨acher, and M. Hut- ter, “Attention-based map encoding for learning generalized legged locomotion,”Science Robotics, vol. 10, no. 105, p. eadv3604, 2025
2025
-
[14]
Elevation mapping for locomotion and navigation using GPU,
T. Miki, L. Wellhausen, R. Grandia, F. Jenelten, T. Homberger, and M. Hutter, “Elevation mapping for locomotion and navigation using GPU,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 2273–2280
2022
-
[15]
Neural scene representation for locomotion on structured terrain,
D. Hoeller, N. Rudin, C. Choy, A. Anandkumar, and M. Hutter, “Neural scene representation for locomotion on structured terrain,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 8667–8674, 2022
2022
-
[16]
Anymal parkour: Learning agile navigation for quadrupedal robots,
D. Hoeller, N. Rudin, D. Sako, and M. Hutter, “Anymal parkour: Learning agile navigation for quadrupedal robots,”Science Robotics, vol. 9, no. 88, p. eadi7566, 2024
2024
-
[17]
Gallant: V oxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains,
Q. Ben, B. Xu, K. Li, F. Jia, W. Zhang, J. Wang, J. Wang, D. Lin, and J. Pang, “Gallant: V oxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains,”arXiv preprint arXiv:2511.14625, 2025
2025
-
[18]
Extreme parkour with legged robots,
X. Cheng, K. Shi, A. Agarwal, and D. Pathak, “Extreme parkour with legged robots,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 443–11 450
2024
-
[19]
FastStair: Learning to Run Up Stairs with Humanoid Robots,
Y . Liu, T. Yu, H. Song, H. Zhu, N. Hu, Y . Hao, X. Yao, X. Zang, H. Chen, and J. Zhao, “FastStair: Learning to Run Up Stairs with Humanoid Robots,”arXiv preprint arXiv:2601.10365, 2026
2026
-
[20]
START: Traversing Sparse Footholds With Terrain Reconstruction,
R. Yu, Q. Wang, H. Li, Z. Jun, Z. Wang, J. Wu, and Q. Zhu, “START: Traversing Sparse Footholds With Terrain Reconstruction,” IEEE Robotics and Automation Letters, vol. 11, no. 2, pp. 2194–2201, 2025
2025
-
[21]
Learning vision-based bipedal locomotion for challenging terrain,
H. Duan, B. Pandit, M. S. Gadde, B. Van Marum, J. Dao, C. Kim, and A. Fern, “Learning vision-based bipedal locomotion for challenging terrain,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 56–62
2024
-
[22]
PIE: Parkour with implicit-explicit learning framework for legged robots,
S. Luo, S. Li, R. Yu, Z. Wang, J. Wu, and Q. Zhu, “PIE: Parkour with implicit-explicit learning framework for legged robots,”IEEE Robotics and Automation Letters, vol. 9, no. 11, pp. 9986–9993, 2024
2024
-
[23]
Humanoid Parkour Learning,
Z. Zhuang, S. Yao, and H. Zhao, “Humanoid Parkour Learning,” in 8th Annual Conference on Robot Learning, 2024
2024
-
[26]
Proximal Policy Optimization Algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” 2017
2017
-
[27]
Learning to walk in minutes using massively parallel deep reinforcement learning,
N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning. PMLR, 2022, pp. 91–100
2022
-
[28]
Warp: A High-performance Python Framework for GPU Simulation and Graphics,
M. Macklin, “Warp: A High-performance Python Framework for GPU Simulation and Graphics,” https://github.com/nvidia/warp, March 2022, NVIDIA GPU Technology Conference (GTC)
2022
-
[29]
AGIBOT X2 Ultra,
AGIBOT, “AGIBOT X2 Ultra,” https://www.agibot.com/products/X2, accessed on 2026-03-25
2026
-
[30]
Unitree H1,
Unitree Robotics, “Unitree H1,” https://www.unitree.com/h1/, ac- cessed on 2026-03-25
2026
-
[31]
MuJoCo: A physics engine for model-based control,
E. Todorov, T. Erez, and Y . Tassa, “MuJoCo: A physics engine for model-based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033
2012
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.