Robotics Research Hub
Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data

Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data

Hu, Songbo, Liao, Qiayuan, Chi, Yufeng, Zakka, Kevin, Shao, Yakun Sophia, Abbeel, Pieter, Sreenath, Koushil · N/A · 2026

manipulation vision reinforcement-learning learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Humanoid robots still struggle to plan contact-rich whole-body manipulation from egocentric RGB and proprioception. Workhorse learns such manipulation from robot-free human demonstrations. A visual planner predicts five-link targets: the poses of the torso, both wrists, and both feet.

Abstract Summary

Humanoid robots still struggle to plan contact-rich whole-body manipulation from egocentric RGB and proprioception. Workhorse learns such manipulation from robot-free human demonstrations. A visual planner predicts five-link targets: the poses of the torso, both wrists, and both feet. A reinforcement-learning whole-body tracker follows them on the robot. Both policies train separately on the same recorded human poses, without retargeting. We augment the training data of each policy to imitate the errors that the other makes at deployment. On a real Unitree G1, Workhorse sorts boxes with its hands and a kick, catches a thrown box, and topples and climbs a suitcase. During box sorting, we show recoveries after a person pushes the robot or takes the box away. In a simulated copy of the demonstration room, the system completes box sorting in 77% of episodes, and in 64% under 40 N.s pushes. With both policies retrained from the same demonstrations, a simulated second humanoid completes box sorting in 83% of episodes without pushes.

Key Points

  • Humanoid robots still struggle to plan contact-rich whole-body manipulation from egocentric RGB and proprioception.
  • Workhorse learns such manipulation from robot-free human demonstrations
  • A visual planner predicts five-link targets: the poses of the torso, both wrists, and both feet
  • A reinforcement-learning whole-body tracker follows them on the robot
  • Both policies train separately on the same recorded human poses, without retargeting

HUMAN DATA COLLECTION

(a) (b) 0 1 $2\text{\,}\mathrm{s}$ audio onset (spectral flux) clap score (c) Both policies learn directly from the human demonstrations, so their quality limits how well the behavior transfers to the robot. This section describes the rig (§ III-A ), the recording protocol (§ III-B ), and the synchronization and processing (§ III-C ).

The protocol splits each session into repetitions, and it perturbs only the object, never the demonstrator.

HUMAN DATA COLLECTION — Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data
Fig. 3: Data collection. (a) The rig: one chest camera with a wearable tracker on its mount, and one wearable tracker on each hand and each foot. (b) The live preview: the robot posed by real-time inverse kinematics, the hand error of this solution, and the visibility of each wearable tracker. (c) Clock synchronization on the three opening claps. The curves are the clap score of the two wearable trackers on the hands and the camera’s audio onset, each scaled to unit peak. Dashed lines mark the matched claps.

HUMANOID SKILL TRANSFER

This section turns the human demonstrations into two policies and runs them on the robot (Fig. 2 ). Both policies share one interface, the recorded poses of five links (§ IV-A ). The visual planner trains on images edited to show the robot and on histories with added drift (§ IV-B ). The whole-body tracker trains on commands that drift and jump as they do at deployment (§ IV-C ).

At deployment, timestamps align the two policies, which run at different rates and with different delays (§ IV-D ).

HUMANOID SKILL TRANSFER — Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data
Fig. 4: One interface, two embodiments. For illustration, we pose each robot by inverse kinematics on the same demonstration and draw it over the demonstrator. (a) Unitree G1, and (b) Unitree H2.

SYSTEM EVALUATION

On a real Unitree G1, three tasks demonstrate long-horizon, contact-rich, and reactive loco-manipulation, and box sorting shows recovery from perturbations (§ V-A ). In simulation, we ablate the design choices (§ V-B ) and retrain both policies for a second humanoid from the same demonstrations (§ V-C ). (a) (b) (c) (d) (e) (f) Fig.

8: Recovery from perturbations during box sorting on the real G1, in one continuous run.

SYSTEM EVALUATION — Workhorse: Learning Robust Whole-Body Humanoid Loco-Manipulation from Human Data
Fig. 5: Image editing and augmentation for the planner. (a) Person-to-robot edit. (b) Photometric augmentation: brightness, lighting gradient, and contrast. (c) Image warp of the drift augmentation for three camera displacements. Each inset shows the camera before (gray) and after (orange) the displacement.

Sources and Demonstrations

Figures are reproduced from Hu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced