Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Trained purely in simulation, one policy runs the full real-world search-grasp-carry-drop task on a low-cost quadruped at 78.3% success with no real-world fine-tuning.

desk verdict A genuinely impressive sim-to-real legged manipulation system, but the headline 78% is best-of-ten on the low-level policy, so treat the number with caution. read the letter →

arxiv 2501.09905 v4 pith:VTAM5EDH submitted 2025-01-17 cs.RO cs.AIcs.CVcs.LG

Haichao Zhang , Haonan Yu , Le Zhao , Andrew Choi , Qinxun Bai , Break Yang , Wei Xu This is my paper · ORCID
classification cs.ROcs.AIcs.CVcs.LG
keywords sim-to-realtransferleggedmobilemanipulationteacher-studentreinforcementlearninglong-horizontaskdecompositionprogressivepolicyexpansionvisuomotorlanguage-conditionedzero-shotdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SLIM's claim is that the entire sim-to-real gap for language-instructed legged manipulation can be crossed without expensive hardware, human demonstrations, or real-world fine-tuning. A single visuomotor policy, trained purely with reinforcement learning in simulation, takes a language instruction like 'drop the yellow cube into the blue basket' and drives a low-cost quadruped through the whole chain: search and approach, grasp, search for the container, carry, and release. In the paper's main evaluation the policy completes the full task in 78.3% of real-world episodes, edging out an expert human teleoperator (75.0%) and doing it faster (43.8 versus 65.5 seconds). The authors attribute this result to a specific architecture: a frozen low-level locomotion controller underneath a high-level policy that was first taught by a 'teacher' given privileged subtask information, then distilled into a 'student' that sees only RGB images, proprioception, and the instruction, with a visual bottleneck forcing the policy to rely on predicted segmentation and depth. If the claim holds, long-horizon mobile manipulation becomes trainable on commodity hardware with nothing but a simulator.

What carries the argument

The load-bearing mechanism is the pairing of a frozen low-level locomotion policy with a high-level teacher-student training loop that ends in a visual bottleneck. The low-level policy, trained with PPO, tracks a 2D velocity command and is what actually moves the quadruped; the high-level teacher, trained with a multi-task variant of SAC, exploits privileged observations — the subtask index $k$ and simulated object states — and is trained via Progressive PEX, a policy-expansion scheme that grows one network instance per subtask so the learner keeps exploring past bottleneck states such as the grasp without losing skills already acquired. The student is trained by a modified SAC objective that replaces the entropy bonus with a KL distillation term against the teacher's action distribution (with the teacher's dispersion fixed to a constant so the student keeps exploring), adds the same task rewards, and supervises the prediction of segmentation and depth maps; at deployment the maps are the policy's only visual input, which the paper argues is the key transfer enabler because it strips the policy down to the minimal visual information the task requires. Around that core sits the enumerated technique suite — arm PID control with small position deltas, stationary manipulation during Grasp and DropInto, arm-mount and object perturbations, texture and background-object randomization, HSV color modeling — each rated critical, somewhat important, or marginal in Table I.

What would settle it

Run the full pipeline from scratch ten times, keeping the low-level policy chosen by a fixed rule (for example, the same seed for every run, or a simulation-only stability score), and count full-task success under the paper's 20-episode protocol; if success for the unselected seeds clusters well below 78.3% ± 5.8%, the headline number is carried by the real-world stability filter rather than by the training method. A cheaper version of the same test is to ask for per-seed reported success on all ten low-level seeds, which the paper does not provide.

Watch

Extended reading notes

Core claim

The central claim is that a hierarchical teacher-student pipeline can solve a long-horizon, language-conditioned manipulation task entirely in simulation and then deploy zero-shot in the real world. The task is decomposed into seven subtasks — Search, MoveTo, Grasp, SearchWithObj, MoveToWithObj, MoveGripperToWithObj, DropInto — and a teacher policy receives privileged inputs: the current subtask index and low-dimensional object states that are visible only when the object is actually in the wrist camera's field of view. The teacher is trained with progressive policy expansion, allocating a fresh network per subtask so that earlier skills are not overwritten while later ones are learned. A student policy is then trained to solve the whole task from the instruction, a stack of RGB frames, and proprioception, combining a KL distillation loss against the teacher's action distribution with the same task rewards, so imitation and environment feedback reinforce each other rather than competing. Sim-to-real transfer is carried by a visual information bottleneck — the student predicts segmentation and depth maps with a U-Net and the downstream policy consumes only those maps — together with dynamics-side choices such as arm PID control, stationary manipulation during grasp, and object perturbations. On a low-cost quadruped with a top-mounted arm and one wrist-mounted RGB camera, the resulting 19-million-parameter system reaches 78.3% ± 5.8% full-task success over 20 episodes per seed across three seeds in the standard lobby layout, versus 75.0% ± 5.0% for an expert teleoperator, in 43.8 versus 65.5 seconds; the same frozen policy holds roughly 78% success in outdoor, carpet, and kitchen scenes, still works in a cluttered out-of-distribution layout, and generalizes to novel object shapes, human interruptions, and repeated consecutive runs. The authors state that, to their knowledge, this is the first end-to-end robotic system to solve long-horizon real-world legged manipulation from sim-to-real RL alone, and they note the system is bounded to flat terrain, four object colors, and a narrow instruction vocabulary.

Load-bearing premise

The result rests on a hand-picked filter: the authors trained ten low-level locomotion policies and kept only the three that shook least in the real world, and every headline success number comes from those three; if the filter rather than the training method is what makes the system succeed, the 78% figure does not describe the pipeline as a whole.

Editorial extensions

If this is right

  • Long-horizon mobile manipulation can be trained with a simulator and commodity hardware alone, so the cost and labor of real-world data collection and demonstration drop out of the loop.
  • The same recipe — privileged teacher with per-subtask networks, then distillation-guided RL into a visual student — becomes a transferable template for other long-horizon tasks with bottleneck states, such as precision insertion.
  • Because deployment perception goes through predicted segmentation and depth maps, perception failures and control failures become separable during debugging, which should speed up future sim-to-real development.
  • With roughly $6–12K of hardware and one wrist-mounted RGB camera, the system defines a low-cost baseline for legged manipulation research.
  • At 43.8 seconds versus 65.5 for expert teleoperation, the autonomous policy is not just a stand-in for human control on this task but the faster actor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's ±5.8% error bar covers only the three low-level seeds selected by real-world stability; the seven unselected seeds are not reported, so a natural extension — and a sharper test of the method — is to publish per-seed full-task success for all ten seeds.
  • The visual bottleneck assumes segmentation and depth carry everything the task needs; an obvious stress test is deployments with a target object that shares the color and shape of a distractor, or with specular lighting, where the U-Net's predictions should degrade first.
  • Because the teacher sees the subtask index and simulated object states, the student inherits a curriculum; the same Progressive-PEX-plus-distillation recipe should port to other tasks whose bottleneck is a precision contact event, such as plug insertion or assembly, where the visual bottleneck would need to encode affordances rather than colors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SLIM, a hierarchical visuomotor system for legged mobile manipulation trained entirely in simulation and deployed zero-shot on a Unitree Go1 with a WidowX-250S arm and a single wrist-mounted RGB camera. A low-level locomotion policy is trained with PPO, a teacher policy uses privileged task decomposition with progressive policy expansion, and a student policy is trained via distillation-guided RL on visual, language, and proprioceptive inputs. The main real-world evaluation reports 78.3% ± 5.8% full-task success across search, move-to, grasp, transport, and drop-into, with episode times about 1.5 times faster than human teleoperation, plus generalization experiments across four additional scenes and 400 total real-world episodes.

Significance. If the reported results hold as stated, SLIM would be a notable advance: it demonstrates end-to-end long-horizon legged manipulation on low-cost hardware without real-world fine-tuning, with a relatively small 19M-parameter policy and no third-party perception modules. The paper's strengths include extensive real-world evaluation (400 episodes), several controlled ablations (No Arm Retract, No Perturbation, No Visual Aug, Distillation Only), and a clear decomposition of the long-horizon task. The sim-to-real techniques are concrete and reproducible in principle, and the authors state an intent to release code. However, the central numerical claim is conditioned on a post-hoc selection of low-level policies, and the human-teleoperation comparison is weaker than the term 'expert' implies; both issues bear directly on the headline conclusions.

major comments (3)
  1. [Appendix E-A, Section VIII-C, Table III] The headline 78.3% full-task success is not an unbiased estimate of the training pipeline's expected performance. Appendix E-A ('Reducing Quadruped Shaking') states that low-level locomotion policies were trained for ten random seeds and only the three 'best ones (less shaking and more stable for the stationary pose)' were used to train high-level policies and for evaluation. Since shaking is directly implicated in grasp and drop failures (Section VIII-D, Section VIII-G), the selection criterion is correlated with task success, and the reported ±5.8% computed over the three selected seeds does not include the variance of the ten-seed selection process. The paper should report results for all ten low-level seeds, or explicitly qualify the abstract and Table III as reporting best-of-ten performance. This point is load-bearing because the central claim is pipeline-level robustness and the comparison with human teleoperation.
  2. [Section VIII-B, Appendix I] The claim that SLIM is comparable to 'expert human teleoperation' is not fully supported by the experimental protocol. Appendix I describes a single human operator allowed one hour of practice, using the same wrist-camera view, and states that the teleoperator was 'less accurate and slower' overall; the Go1 over-current protection failures were ignored and episodes restarted. This is a reasonable reference point, but it is not an expert-teleoperation benchmark. The comparison in Table III is still informative, but the abstract and Section VIII-D should be reworded to say 'a practiced human teleoperator' rather than 'expert human teleoperation.'
  3. [Section VIII-C, Fig. 11] The statement in Section VIII-C that 'we run the entire training pipeline across three different random seeds for all methods' conflicts with the low-level seed selection in Appendix E-A. The three seeds used for all methods are not three independent realizations of the full pipeline; they are the best three of ten low-level seeds, chosen by a criterion that is not orthogonal to downstream task performance. Consequently, the simulation-versus-real comparison in Fig. 11 is also conditional on this selection. Please clarify the number of independent training repetitions and the role of seed selection in all reported averages.
minor comments (5)
  1. [Section II-B] The text says 'as apposed to some of the existing methods' but should be 'as opposed to.'
  2. [Appendix B-B] The term 'sationary bit' appears in the stationary-bit paragraph; it should be 'stationary bit.'
  3. [Section IV] The phrase 'Given an subtask id k' should be 'Given a subtask id k.'
  4. [Appendix C] The code URL is given as https://github.com/placeholder_url; if the code release is planned, the placeholder should be replaced with a real URL before publication.
  5. [Appendix E-A] The description of the online adaptation module in Appendix B-C and the low-level policy input design would benefit from a brief explanation of why arm joint velocities are excluded; the current text states the empirical finding but does not give the delay evidence in the main flow.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the headline success is an independent real-world measurement, and the self-cited building blocks (PEX, SACLite) are backed by in-paper ablations and external evaluations.

full rationale

SLIM's central claim is an empirical sim-to-real result, not a derivation. The student objective (Eq. 2) combines a distillation loss to the teacher with an RL loss using the same task rewards; this defines the training procedure, but it does not force the reported 78.3% real-world full-task success, which is measured by physical success criteria (object grasped, transported, and dropped into the basket) across 360 lobby episodes plus 40 additional scene episodes. The teacher-student pipeline, task decomposition, and Progressive PEX are design components with internal evidence: Appendix J-B compares Progressive PEX against a standard network on this task, and Table III compares SLIM against Distillation Only and other ablations, so the load-bearing comparisons do not reduce to the citations. The paper does cite prior work by overlapping authors — PEX [66] for policy expansion, SACLite [63] for removing the entropy reward, and PaCo [51] for multi-task SAC; these are building-block choices, but they are not invoked as unverified uniqueness theorems or as the sole support for the main result, and both are externally published methods with their own evaluations. The most notable caveat is Appendix E-A's selection of three of ten low-level seeds by real-world 'less shaking and more stable for the stationary pose' before high-level training and evaluation; this can make the headline optimistic relative to a randomly seeded pipeline and inflates the reported mean relative to the full seed distribution, but it is an evaluation-selection bias rather than a circular derivation, because full-task success is measured independently and is not defined in terms of the selection criterion. No equation in the paper is equivalent to its inputs by construction, and no prediction is merely a renamed fitted parameter.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced. The central dependence is on the assumed fidelity of simulation, the sufficiency of the visual bottleneck, and the hand-designed task decomposition and rewards. The most consequential free parameter is the post-hoc selection of low-level seeds, which directly affects the reported success rate.

free parameters (6)
  • Low-level seed selection = 3 of 10 seeds chosen for least shaking and stable stationary pose
    Reported success rates depend on high-level policies trained only on the 3 selected low-level seeds; post-hoc selection can inflate performance.
  • Distillation weight alpha = 0.01
    Controls the trade-off between RL and distillation in student training (Eq. 2); affects final policy quality.
  • Teacher action modal dispersion sigma = 0.05
    Fixed std used in distillation to encourage exploration while preserving teacher mode.
  • Object perturbation radius r_perturb = 10 cm
    Radius for teleporting target objects during training to avoid memorized deterministic trajectories.
  • Arm control noise = ±0.02, increased to ±0.05 after 3M teacher iterations
    Noise added to arm joint targets to improve robustness to real control noise.
  • Subtask definitions and success thresholds = K=7 plus Idle; e.g., Grasp succeeds when object lifted 5cm for 2 timesteps
    Hand-defined task decomposition used as privileged information for teacher; defines reward boundaries and behavior priors.
assumptions (6)
  • domain assumption MuJoCo simulation with the described domain randomization and reward design is a sufficiently accurate model of the real Go1+WidowX system for zero-shot transfer.
    The entire system is trained in simulation and deployed without real-world fine-tuning; this transfer is the core enabling assumption.
  • domain assumption The visual information bottleneck (predicted segmentation and depth from RGB) preserves all task-relevant information needed for the policy.
    The student policy only receives the predicted maps as visual input; if the bottleneck discards critical cues (e.g., object identity or color), the policy cannot succeed.
  • domain assumption The hand-defined subtask decomposition and success conditions are valid and provide an effective curriculum for the teacher.
    Teacher training relies on privileged subtask ids and boundary conditions; incorrect decomposition would prevent learning the long-horizon task.
  • domain assumption The reward functions (sparse subtask success, distance shaping, keep-grasping penalty, arm-retract) induce the intended safe behaviors in simulation and real.
    RL optimization targets these rewards; they define what 'success' means and shape the policy's real-world behavior.
  • domain assumption Behavior priors (stationary manipulation, rotational search) do not prevent task completion and are consistently enforced in the student via the stationary bit.
    These priors are hard-coded by zeroing action means; if they conflict with task requirements, the policy would be suboptimal.
  • standard math Standard RL convergence and neural network training assumptions (SAC, PPO, U-Net, etc.) hold as typical in the literature.
    The paper relies on standard deep RL and supervised learning methodology.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learning." pith.science (2026). https://pith.science/paper/VTAM5EDH

@misc{pith2026250109905,
  author       = {Pith},
  title        = {Pith review of: SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VTAM5EDH}},
  note         = {Machine review of arXiv:2501.09905}
}
read the original abstract

We present a low-cost legged mobile manipulation system that solves long-horizon real-world tasks, trained by reinforcement learning purely in simulation. This system is made possible by 1) a hierarchical design of a high-level policy for visual-mobile manipulation following task instructions, and a low-level quadruped locomotion policy, 2) a teacher and student training pipeline for the high level, which trains a teacher to tackle long-horizon tasks using privileged task decomposition and target object information, and further trains a student for visual-mobile manipulation via RL guided by the teacher's behavior, and 3) a suite of techniques for minimizing the sim-to-real gap. In contrast to many previous works that use high-end equipments, our system demonstrates effective performance with more accessible hardware -- specifically, a Unitree Go1 quadruped, a WidowX-250S arm, and a single wrist-mounted RGB camera -- despite the increased challenges of sim-to-real transfer. Trained fully in simulation, a single policy autonomously solves long-horizon tasks involving search, move to, grasp, transport, and drop into, achieving nearly 80% real-world success. This performance is comparable to that of expert human teleoperation on the same tasks while the robot is more efficient, operating at about 1.5x the speed of the teleoperation. Finally, we perform extensive ablations on key techniques for efficient RL training and effective sim-to-real transfer, and demonstrate effective deployment across diverse indoor and outdoor scenes under various lighting conditions.

Figures

Figures reproduced from arXiv: 2501.09905 by the authors.

Figure 1
Figure 1. SLIM in Real. Snapshots of the same SLIM policy deployed in diverse real-world scenes, featuring significant variations in terrain, background, distractors, and other environmental factors. These scenes are not available in simulation training. The subtask annotations at the bottom are added for understanding task progress and are not part of the input to the system. Abstract—We present a low-cost legged mobile mani… view at source ↗
Figure 2
Figure 2. Hierarchical Framework and Visuomotor Policy Pipeline (Left). Given language instruction and sensor inputs, the high-level policy generates a set of two control signals: 1) the arm and gripper control signals, and 2) the locomotion command. The arm control signals are directly passed to the arm driver, and the locomotion command is passed to the low-level policy to control the leg joints of the quadruped. Low-Level … view at source ↗
Figure 3
Figure 3. Teacher Policy Network Structure. The full teacher network Π is a set of structurally identical networks {π k} K k=1 gated by the subtask id k. On the left, each individual teacher policy π k takes a set of privileged and proprioceptive input, flattening and concatenating them before passing it through an MLP. On the right, we show the full teacher network Π. Given an subtask id k, the k-th individual teacher policy… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Student Architecture Overview. Red dashed lines represent data flows that only exist in simulation training. Solid lines denote the inference process of the student policy. “background”. We use a U-Net [46] to predict the segmentation and depth maps. This is achieved b…
Figure 5
Figure 5. Figure 5: Task Decomposition as Privileged Information (for teacher only). Given a task, we decompose it into a sequence of subtasks. As shown above, the task of finding an object A, grasping it, and then dropping into an object B can be decomposed into a sequence of subtasks. I…
Figure 6
Figure 6. Figure 6: Qualitative Overview of the Robot System. Intel i9-12900H CPU, NVIDIA RTX 3070Ti laptop GPU, and 16GB of RAM. To minimize latency, we execute high- and low￾level inference asynchronously (Appendix F-A) at a frequency of 10 and 50 Hz, respectively. The outputs from the …
Figure 7
Figure 7. Figure 7: Scene Spatial Layouts in Lobby Environment. Standard: we put the robot in the center of a 2m×2m square space and objects including objects to be grasped and containers to drop the graspable objects into on the four corners. The instructions and object placements vary a…
Figure 8
Figure 8. Figure 8: Policy Behaviors With and Without Arm Retract. The arm controlled by the policy trained without the arm-retract reward is over-extended, leading to more arm shake, cube drops and a higher chance of search failures. Secondly, right after grasp, the gripper and the camer…
Figure 9
Figure 9. Figure 9: The same SLIM Policy across Scene Variations. Left: images showing the differences for evaluation. Right: the average full task success rate across 10 real-world deployment episodes for each of the scenes shown on the left. to complete the full task, which is about 1.5…
Figure 10
Figure 10. Figure 10: Generalization Behaviors of SLIM Policy. Novel Object: grasping an object with a novel shape that is out of the training distribution. Re-Grasping: a human interrupts the task progress by removing the cube from the gripper and toss it to the ground. The SLIM policy wi…
Figure 11
Figure 11. Figure 11: Full Task Sim and Real Success Rate Comparison. All success rates are the averaged success rate across three random seeds. case, the low-level policy shook and moved backward, causing the cube to be just out of reach. DropInto sometimes failed because of either early …
Figure 1
Figure 1. Figure 1: We also provide demo videos in the supplementary file. [PITH_FULL_IMAGE:figures/full_fig_p024_1.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Visual Signals Mislead: A Mechanistic Study of Attribute Hallucination in Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A null-image diagnostic shows VLM attribute hallucinations track the visual logit margin, not the language prior, and a routed Calib/Abstain/Adapt framework reduces them.

Reference graph

Works this paper leans on

99 extracted references · 63 canonical work pages · cited by 1 Pith paper

  1. [1]

    Legged locomotion in challenging terrains using egocentric vision

    Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning, 2022

    2022

  2. [2]

    Vision-aided dynamic quadrupedal locomotion on discrete terrain using motion libraries

    Ayush Agrawal, Shuxiao Chen, Akshara Rai, and Koushil Sreenath. Vision-aided dynamic quadrupedal locomotion on discrete terrain using motion libraries. In IEEE International Conference on Robotics and Automation , 2022

    2022

  3. [3]

    On the role of the action space in robot manipulation learning and sim-to-real transfer

    Elie Aljalbout, Felix Frank, Maximilian Karl, and Patrick van der Smagt. On the role of the action space in robot manipulation learning and sim-to-real transfer. IEEE Robotics and Automation Letters , 9(6):5895–5902, June 2024

    2024

  4. [4]

    CoRR, arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xi- aoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.π0: A vis...

    arXiv

  5. [5]

    RT-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang- Huei Lee, Sergey Levine, Yao Lu, Utsav Mall...

    arXiv 2023

  6. [6]

    Ken Caluwaerts, Atil Iscen, J. Chase Kew, Wenhao Yu, Tingnan Zhang, Daniel Freeman, Kuang-Huei Lee, Lisa Lee, Stefano Saliceti, Vincent Zhuang, Nathan Batchelor, Steven Bohez, Federico Casarini, Jose Enrique Chen, Omar Cortes, Erwin Coumans, Adil Dostmohamed, Gabriel Dulac-Arnold, Alejandro Escontrela, Erik Frey, Roland Hafner, Deepali Jain, Bauyrjan Jyen...

    arXiv 2023

  7. [7]

    Learning to explore using active neural SLAM

    Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural SLAM. In International Conference on Learning Representations , 2020

    2020

  8. [8]

    Extreme parkour with legged robots

    Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. CoRR, arXiv:2309.14341, 2023

    arXiv 2023

Show all 99 references
  1. [9]

    Dario Bellicoso, Fabian Jenelten, P ´eter Fankhauser, Christian Gehring, Jemin Hwangbo, and Marco Hut- ter

    C. Dario Bellicoso, Fabian Jenelten, P ´eter Fankhauser, Christian Gehring, Jemin Hwangbo, and Marco Hut- ter. Dynamic locomotion and whole-body control for quadrupedal robots. In International Conference on Intelligent Robots and Systems , 2017

    2017

  2. [10]

    Google scanned objects: A high-quality dataset of 3D scanned household items

    Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In IEEE International Conference on Robotics and Automation, 2022

    2022

  3. [11]

    Playful DoggyBot: Learning agile and precise quadrupedal locomotion

    Xin Duan, Ziwen Zhuang, Hang Zhao, and Soeren Schw- ertfeger. Playful DoggyBot: Learning agile and precise quadrupedal locomotion. CoRR, arXiv:2409.19920, 2024

    2024 arXiv

  4. [12]

    Secant: Self-expert cloning for zero-shot generalization of visual policies

    Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Anima Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In International Conference on Machine Learn- ing, 2021

    2021

  5. [13]

    Dario Bellicoso, Takahiro Miki, and Marco Hutter

    Peter Fankhauser, Marko Bjelonic, C. Dario Bellicoso, Takahiro Miki, and Marco Hutter. Robust rough-terrain lo- comotion with a quadrupedal robot. In IEEE International Conference on Robotics and Automation , 2018

    2018

  6. [14]

    GenLoco: Generalized locomotion controllers for quadrupedal robots

    Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Xhitao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath, and Sergey Levine. GenLoco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning , 2022

    2022

  7. [15]

    Minimizing energy consumption leads to the emergence of gaits in legged robots

    Zipeng Fu, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Minimizing energy consumption leads to the emergence of gaits in legged robots. In Conference on Robot Learning, 2021

    2021

  8. [16]

    Deep whole-body control: Learning a unified policy for manip- ulation and locomotion

    Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: Learning a unified policy for manip- ulation and locomotion. In Conference on Robot Learning, 2022

    2022

  9. [17]

    Coupling vision and proprioception for navigation of legged robots

    Zipeng Fu, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Jitendra Malik, and Deepak Pathak. Coupling vision and proprioception for navigation of legged robots. In IEEE Conference on Computer Vision and Pattern Recognition , 2022

    2022

  10. [18]

    Zhao, and Chelsea Finn

    Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low- cost whole-body teleoperation. In Conference on Robot Learning (CoRL), 2024

    2024

  11. [19]

    Hoepflinger, and Roland Siegwart

    Christian Gehring, Stelian Coros, Marco Hutter, Michael Bloesch, Markus A. Hoepflinger, and Roland Siegwart. Control of dynamic gaits for a quadrupedal robot. In IEEE International Conference on Robotics and Automation , 2013

    2013

  12. [20]

    UMI-on-legs: Making manipulation policies mobile with a manipulation-centric whole-body controller

    Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran Song. UMI-on-legs: Making manipulation policies mobile with a manipulation-centric whole-body controller. In Conference on Robot Learning , 2024

    2024

  13. [21]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , 2018

    2018

  14. [22]

    Deep hierarchical planning from pixels

    Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. In Advances in Neural Information Processing Systems , 2022

    2022

  15. [23]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  16. [24]

    Manipulator as a tail: Promoting dynamic stability for legged locomo- tion

    Huang Huang, Antonio Loquercio, Ashish Kumar, Neerja Thakkar, Ken Goldberg, and Jitendra Malik. Manipulator as a tail: Promoting dynamic stability for legged locomo- tion. In IEEE International Conference on Robotics and Automation, 2024

    2024

  17. [25]

    Depth map artefacts reduction: a review

    Mostafa Mahmoud Ibrahim, Qiong Liu, Rizwan Khan, Jingyu Yang, Ehsan Adeli, and You Yang. Depth map artefacts reduction: a review. IET Image Processing , 14: 2630–2644, 2020

    2020

  18. [26]

    Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization

    Chieko Sarah Imai, Minghao Zhang, Yuchen Zhang, Marcin Kierebinski, Ruihan Yang, Yuzhe Qin, and Xi- aolong Wang. Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022

    2022

  19. [27]

    Provably stabi- lizing controllers for quadrupedal robot locomotion on dynamic rigid platforms

    Amir Iqbal, Yuan Gao, and Yan Gu. Provably stabi- lizing controllers for quadrupedal robot locomotion on dynamic rigid platforms. IEEE/ASME Transactions on Mechatronics, 25(4):2035–2044, 2020

    2020

  20. [28]

    Sim- to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks

    Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim- to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In IEEE Confe...

    2019

  21. [29]

    Learning whole-body manip- ulation for quadrupedal robot

    Seunghun Jeon, Moonkyu Jung, Suyoung Choi, Beomjoon Kim, and Jemin Hwangbo. Learning whole-body manip- ulation for quadrupedal robot. CoRR, arXiv:2308.16820, 2023

    2023 arXiv

  22. [30]

    DribbleBot: Dynamic legged manipulation in the wild

    Yandong Ji, Gabriel B Margolis, and Pulkit Agrawal. DribbleBot: Dynamic legged manipulation in the wild. In IEEE International Conference on Robotics and Automation, 2023

    2023

  23. [31]

    RMA: Rapid motor adaptation for legged robots

    Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: Rapid motor adaptation for legged robots. In Robotics: Science and Systems , 2021

    2021

  24. [32]

    Dragan, and Ken Goldberg

    Michael Laskey, Jonathan Lee, Roy Fox, Anca D. Dragan, and Ken Goldberg. DART: noise injection for robust imitation learning. In Conference on Robot Learning , 2017

    2017

  25. [33]

    LocoMan: Advancing versatile quadrupedal dexterity with lightweight loco-manipulators

    Changyi Lin, Xingyu Liu, Yuxiang Yang, Yaru Niu, Wenhao Yu, Tingnan Zhang, Jie Tan, Byron Boots, and Ding Zhao. LocoMan: Advancing versatile quadrupedal dexterity with lightweight loco-manipulators. CoRR, arXiv:2403.18197, 2024

    2024 arXiv

  26. [34]

    Visual whole-body control for legged loco-manipulation

    Minghuan Liu, Zixuan Chen, Xuxin Cheng, Yandong Ji, Ruihan Yang, and Xiaolong Wang. Visual whole-body control for legged loco-manipulation. In Conference on Robot Learning, 2024

    2024

  27. [35]

    The HIM solution for legged locomotion: Minimal sensors, efficient learning, and substantial agility

    Junfeng Long, ZiRui Wang, Quanyi Li, Liu Cao, Jiawei Gao, and Jiangmiao Pang. The HIM solution for legged locomotion: Minimal sensors, efficient learning, and substantial agility. In International Conference on Learning Representations, 2024

    2024

  28. [36]

    Understand- ing and preventing capacity loss in reinforcement learning

    Clare Lyle, Mark Rowland, and Will Dabney. Understand- ing and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations , 2022

    2022

  29. [37]

    Under- standing plasticity in neural networks

    Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity in neural networks. In International Conference on Machine Learning , 2023

    2023

  30. [38]

    Combining learning-based loco- motion policy with model-based manipulation for legged mobile manipulators

    Yuntao Ma, Farbod Farshidian, Takahiro Miki, Joonho Lee, and Marco Hutter. Combining learning-based loco- motion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377–2384, 2022

    2022

  31. [39]

    Rapid locomotion via reinforcement learning

    Gabriel Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. In Robotics: Science and Systems , 2022

    2022

  32. [40]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior

    Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning , 2022

    2022

  33. [41]

    Learning to see physical properties with active sensing motor policies

    Gabriel B Margolis, Xiang Fu, Yandong Ji, and Pulkit Agrawal. Learning to see physical properties with active sensing motor policies. In Conference on Robot Learning, 2023

    2023

  34. [42]

    RoboDuet: A framework affording mobile-manipulation and cross-embodiment

    Guoping Pan, Qingwei Ben, Zhecheng Yuan, Guangqi Jiang, Yandong Ji, Jiangmiao Pang, Houde Liu, and Huazhe Xu. RoboDuet: A framework affording mobile-manipulation and cross-embodiment. CoRR, arXiv:2403.17367, 2024

    2024 arXiv

  35. [43]

    Courville

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence , 2018

    2018

  36. [44]

    Learning gener- alizable feature fields for mobile manipulation

    Ri-Zhao Qiu, Yafei Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, and Xiaolong Wang. Learning gener- alizable feature fields for mobile manipulation. CoRR, arXiv:2403.07563, 2024

    2024 arXiv

  37. [45]

    SplatSim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting

    Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, and Abhisesh Silwal. SplatSim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting. CoRR, arXiv:2409.10161, 2024

    2024 arXiv

  38. [46]

    U-Net: Convolutional networks for biomedical image seg- mentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Computer- Assisted Intervention, 2015

    2015

  39. [47]

    Efficient reductions for imitation learning

    Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics , 2010

    2010

  40. [48]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, 2021

    2021

  41. [49]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, arXiv:1707.06347, 2017

    2017 arXiv

  42. [50]

    Versatile multicontact planning and control for legged loco-manipulation

    Jean-Pierre Sleiman, Farbod Farshidian, and Marco Hutter. Versatile multicontact planning and control for legged loco-manipulation. Science Robotics , 8(81):eadg5014, 2023

    2023

  43. [51]

    PaCo: Parameter-compositional multi-task re- inforcement learning

    Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. PaCo: Parameter-compositional multi-task re- inforcement learning. In Advances in Neural Information Processing Systems, 2022

    2022

  44. [52]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017

    2017

  45. [53]

    MuJoCo: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems , 2012

    2012

  46. [54]

    Lempitsky

    Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, arXiv:1607.08022, 2016

    2016 arXiv

  47. [55]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017

    2017

  48. [56]

    Tidybot++: An open-source holonomic mobile manipulator for robot learning

    Jimmy Wu, William Chong, Robert Holmberg, Aaditya Prasad, Yihuai Gao, Oussama Khatib, Shuran Song, Szymon Rusinkiewicz, and Jeannette Bohg. Tidybot++: An open-source holonomic mobile manipulator for robot learning. In Conference on Robot Learning , 2024

    2024

  49. [57]

    Helpful DoggyBot: Open-world object fetching using legged robots and vision-language models

    Qi Wu, Zipeng Fu, Xuxin Cheng, Xiaolong Wang, and Chelsea Finn. Helpful DoggyBot: Open-world object fetching using legged robots and vision-language models. CoRR, arXiv:2410.00231, 2024

    2024 arXiv

  50. [58]

    Generalized animal imitator: Agile locomotion with versatile motion prior

    Ruihan Yang, Zhuoqun Chen, Jianhan Ma, Chongyi Zheng, Yiyu Chen, Quan Nguyen, and Xiaolong Wang. Generalized animal imitator: Agile locomotion with versatile motion prior. In Conference on Robot Learning , 2024

    2024

  51. [59]

    Image augmentation is all you need: Regularizing deep reinforce- ment learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforce- ment learning from pixels. In International Conference on Learning Representations , 2021

    2021

  52. [60]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations , 2022

    2022

  53. [61]

    ASC: Adap- tive Skill Coordination for Robotic Mobile Manipulation

    Naoki Yokoyama, Alexander William Clegg, Joanne Truong, Eric Undersander, Jimmy Yang, Sergio Arnaud, Sehoon Ha, Dhruv Batra, and Akshara Rai. ASC: Adap- tive Skill Coordination for Robotic Mobile Manipulation. IEEE Robotics and Automation Letters , 2023

    2023

  54. [62]

    Learning visual parkour from generated images

    Alan Yu, Ge Yang, Ran Choi, Yajvan Ravan, John Leonard, and Phillip Isola. Learning visual parkour from generated images. In Conference on Robot Learning , 2024

    2024

  55. [63]

    Do you need the entropy reward (in practice)? CoRR, arXiv:2201.12434, 2022

    Haonan Yu, Haichao Zhang, and Wei Xu. Do you need the entropy reward (in practice)? CoRR, arXiv:2201.12434, 2022

    2022 arXiv

  56. [64]

    Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2019

    2019

  57. [65]

    NIS-SLAM: Neural implicit semantic RGB-D SLAM for 3d consistent scene understanding

    Hongjia Zhai, Gan Huang, Qirui Hu, Guanglin Li, Hujun Bao, and Guofeng Zhang. NIS-SLAM: Neural implicit semantic RGB-D SLAM for 3d consistent scene understanding. IEEE Transactions on Visualization and Computer Graphics, pages 1–11, 2024

    2024

  58. [66]

    Policy expan- sion for bridging offline-to-online reinforcement learning

    Haichao Zhang, Wei Xu, and Haonan Yu. Policy expan- sion for bridging offline-to-online reinforcement learning. In International Conference on Learning Representations , 2023

    2023

  59. [67]

    GAMMA: Graspability-aware mobile manipulation policy learning based on online grasping pose fusion

    Jiazhao Zhang, Nandiraju Gireesh, Jilong Wang, Xi- aomeng Fang, Chaoyi Xu, and Weiguang Chen. GAMMA: Graspability-aware mobile manipulation policy learning based on online grasping pose fusion. In IEEE Interna- tional Conference on Robotics and Automation , 2024

    2024

  60. [68]

    Learning to open and traverse doors with a legged manipulator

    Mike Zhang, Yuntao Ma, Takahiro Miki, and Marco Hutter. Learning to open and traverse doors with a legged manipulator. In Conference on Robot Learning , 2024

    2024

  61. [69]

    Robot parkour learning

    Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, S ¨oren Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In Conference on Robot Learning, 2023. TABLE OF CONTENTS FOR APPENDIX Appendix A: RL Formulation 16 A-A Subtask Boundaries, Rewards and Be- ...

    2023

  62. [70]

    S(Search): succeeds when the target object is in the view of the wrist-mounted RGB camera

  63. [71]

    S(MoveTo): succeeds when the target object enters the workspace of the robot within its arm reach

  64. [72]

    S(Grasp): succeeds when the target object is in the gripper and lifted 5cm above the ground for at least two timesteps

  65. [73]

    S(SearchWithObj): succeeds when the target object is grasped and the target basket enters the view of the wrist- mounted camera

  66. [74]

    S(MoveToWithObj): succeeds when the target object is grasped and the target basket enters a workspace of the robot within its arm reach

  67. [75]

    S(MoveGripperToWithObj): succeeds when the target object is grasped and the gripper has moved to the top of the target basket

  68. [76]

    swing” phase where smaller movements in the X-Y plane will be penalized; schedule 0 represents the “contact

    S(DropInto): succeeds when the target object enters the target basket. Rewards. We summarize the description of the subtask rewards in Table IV. The task reward represents the sparse success (+1) or failure ( −1) reward. For timeout, we use a reward of 0. Apart from the sparse...

  69. [77]

    Due to the simplicity of our language instructions, we directly encode each instruction into a sequence of byte tokens

    Language. Due to the simplicity of our language instructions, we directly encode each instruction into a sequence of byte tokens. We set the max sequence length to be 100, padding shorter sequences with zeros while truncating longer sequences beyond this limit. Then each byte ...

  70. [78]

    We use a temporal stack of N = 5

    Proprioceptive state. We use a temporal stack of N = 5. Different proprioceptive observations are simply flattened and concatenated together to produce a 1D robot state vector

  71. [79]

    We use a temporal stack of N = 5 RGB images for vision

    Vision. We use a temporal stack of N = 5 RGB images for vision. The images are stacked along the channel dimension and fed to a U-Net encoder containing 6 convolutional blocks. Each block has a convolutional layer (f ilters, kernelsize, strides, padding) ≜ (c, k, s, p), an ins...

  72. [80]

    Low Level

    the standard proprioceptive state sleg obtained from the environment, 2) the locomotion command c from an external command issuer ( e.g., the second type of input is crucial to make the low-level policy useful in downstream tasks where locomotion is involved as a basic skill f...

    2000

  73. [81]

    This could delay the low-level policy (50 HZ) considerably and make the robot falter, if the two levels are executed in a synchronous mode

    Asynchronous low-high inference: most of the model latency comes from the high-level inference, which takes around 20 ms, but can sometimes peak to 40 ms. This could delay the low-level policy (50 HZ) considerably and make the robot falter, if the two levels are executed in a ...

  74. [82]

    However, we find the Wi-Fi connection between Go1 and the laptop to be unstable, sometimes spiking to a latency of 100 ms, which is unacceptable for low-level motor control

    Ethernet cable connection to the robot: Wi-Fi connection is more convenient than cable. However, we find the Wi-Fi connection between Go1 and the laptop to be unstable, sometimes spiking to a latency of 100 ms, which is unacceptable for low-level motor control. It is possible ...

  75. [83]

    Camera observation delay is sampled from (30, 50) ms uniformly per step with observations taken at 60 HZ

    Sensor delay randomizations: the mean of motor sensor reading delays is sampled within (3, 13) ms every episode, with ± 2ms variation sampled uniformly per control step. Camera observation delay is sampled from (30, 50) ms uniformly per step with observations taken at 60 HZ. L...

  76. [84]

    We did not observe any performance drop with the added delay, and simply train the high level without inference delay in this work

    High-level inference delay: we trained policies without high- level inference delay, and compared the performance with adding additional test-time delay. We did not observe any performance drop with the added delay, and simply train the high level without inference delay in th...

  77. [85]

    Similarly, we trained and deployed a standalone quadruped policy, without the mounted arm

    We first addressed all sim-to-real gaps of the arm alone by training and deploying a tabletop policy. Similarly, we trained and deployed a standalone quadruped policy, without the mounted arm. Only after the two standalone policies were transferred well, we started training an...

  78. [86]

    This strategy reduced damages to the arm motor gears significantly

    We used PID control for the arm (Section E-A), with a reasonably small delta action z = 0.05, resulting in much smoother and more reliable arm movement. This strategy reduced damages to the arm motor gears significantly

  79. [87]

    (Section E-A), though making simulation training more difficult, actually allowed the policy to adapt to the real world more easily, resulting in very accurate grasping executions

    The randomizations of motor strength, payload, center of mass, arm mounting position, etc. (Section E-A), though making simulation training more difficult, actually allowed the policy to adapt to the real world more easily, resulting in very accurate grasping executions. Thus ...

  80. [88]

    A.16) to make sure that the high-level inference did not prevent the robot from receiving closed-loop low-level commands

    As high-level inference can sometimes take over 40ms to finish, we adopted asynchronous low-high inference (Fig. A.16) to make sure that the high-level inference did not prevent the robot from receiving closed-loop low-level commands. Otherwise, the quadruped could topple over...

  81. [89]

    When latency spikes, especially when the deployment machine suddenly freezes or the network disconnects, our driver on Go1 will automatically stop the quadruped

    We also enabled deployment termination with a single button press. When latency spikes, especially when the deployment machine suddenly freezes or the network disconnects, our driver on Go1 will automatically stop the quadruped. This preventative stop was done using a scripted...

  82. [90]

    This was done for all baseline methods and SLIM

    To prevent the robot from wandering off and bumping into surroundings, during training we incorporated a rotational search behavior as explained in Section A-A. This was done for all baseline methods and SLIM

  83. [91]

    Later when the policy became more reliable in real, we removed the leash

    We attached a leash to the quadruped to prevent it falling down due to any reason. Later when the policy became more reliable in real, we removed the leash

  84. [92]

    6) was designed to break easily, sacrificing itself to protect the camera in the event of a side fall

    Our 3D-printed camera mount (Fig. 6) was designed to break easily, sacrificing itself to protect the camera in the event of a side fall

  85. [93]

    A very hot Go1 motor will result in unpredictable, dangerous locomotion behaviors

    During deployment, we frequently monitored the temper- ature of Go1’s motors and took a break if necessary. A very hot Go1 motor will result in unpredictable, dangerous locomotion behaviors. APPENDIX G REAL -WORLD EXPERIMENT DETAILS A. Containers and Graspable Objects We use c...

  86. [94]

    A.18: Policy Trajectory Visualization

    For each method, we run 20 episodes with different initial scene states; Fig. A.18: Policy Trajectory Visualization. For visualization purpose, we removed background objects. We also visualize the EE trajectories. Trajectories corresponding to different subtasks are rendered w...

  87. [95]

    We apply a time limit of 90 seconds for each evaluation episode

  88. [96]

    7, as it is easy to ensure accuracy of the scene setup in terms of spatial positions

    To ensure the repeatability of the scene setup, we use the Standard spatial layout as shown in Fig. 7, as it is easy to ensure accuracy of the scene setup in terms of spatial positions. In practice, we mark the positions for the robot at the center of the square. Similarly, we...

  89. [97]

    Drop the yellow cube into the red basket

    We follow a scene initialization procedure as detailed in Table IX across the 20 episodes to ensure good coverage of the positions of the objects, the target object and basket in the language instruction, and the colors of the objects. APPENDIX H VISUALIZATIONS A. Visualizatio...

  90. [98]

    This is sometimes due to not fully grasping the cube and the cube slipping out of the fingers during movement, leaving no time for searching and grasping again

    The most common failure mode for teleoperation is task timeout. This is sometimes due to not fully grasping the cube and the cube slipping out of the fingers during movement, leaving no time for searching and grasping again. Failing to grasp fully is in turn due to either stop...

  91. [99]

    This pose either overloads the arm motors, or if not, flips the camera upside down, making the operator difficult to perceive the environment

    The arm can have a twisted pose due to IK near singularities. This pose either overloads the arm motors, or if not, flips the camera upside down, making the operator difficult to perceive the environment. Overall, human teleoperation is less accurate and slower than an RL poli...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.