Robotics Research Hub
GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

Featured

Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan · NVIDIA, UCLA · 2026

sim-to-real reinforcement-learning vision manipulation humanoid
PDF arXiv Code (275★)

Framework

Python

License

N/A

Stars

275

Summary

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation.

Abstract Summary

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84% real-world success on diverse object pick-up and 90% success on stair-climbing.

Key Points

  • Demonstrates humanoid robot control on real hardware
  • Addresses dexterous manipulation challenges
  • Uses simulation or synthetic data for training
  • Leverages 3D geometric reasoning for perception
  • Enables complex whole-body climbing behaviors
  • Learns multimodal motion skills from human demonstrations

Method

Given a 3D object asset $\mathcal{M}^{\mathcal{O}}$, GRAIL produces humanoid loco-manipulation demonstrations comprising humanoid kinematic motion $\{\bm{\Theta}^{\mathcal{R}}_{t}\}_{t=1}^{T}$, object kinematic motion $\{\bm{\Theta}^{\mathcal{O}}_{t}\}_{t=1}^{T}$, and robot actions $\{\bm{a}^{\mathcal{R}}_{t}\}_{t=1}^{T}$. Our data generation pipeline proceeds in three stages.

Method — GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Figure 2 : Asset-Conditioned 4D HOI Generation. Given a 3D object asset, we render a fully specified 3D scene with a character prefitted to the target humanoid and known camera parameters, synthesize a static-camera interaction video via a VFM conditioned on the rendered frame, and reconstruct metric 4D human-object motion by jointly refining initial human and object trajectory estimates with keypoint, depth, and contact losses anchored to the privileged 3D configuration.

Results

Our experiments cover the three stages of the GRAIL pipeline. We first evaluate whether the generated 4D HOI sequences are more physically executable than existing generation baselines. We then ask whether these 4D HOI sequences can be converted into task-general loco-manipulation policies at scale, rather than only replayed through per-sequence tracking.

Finally, we demonstrate the practical value of the generated data by deploying egocentric visual policies on the real robot for autonomous loco-manipulation. Setup.

Results — GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Figure 3 : Task-General Tracking via Complementary Controller Adaptation. Retargeted 4D HOI trajectories are converted into robot-action data by adapting different parts of a pretrained whole-body controller. Object-manipulation trajectories are used to train an object-aware adaptor $\pi_{\phi}$ while the controller remains frozen: the adaptor observes proprioception $\bm{s}_{t}$ and an object reference $\bm{o}_{t}$, injects a latent residual $\Delta\bm{z}_{t}$, and emits hand actions $\bm{a}^{\mathrm{hand}}_{t}$. Terrain traversal and chair-sitting trajectories are used to fine-tune the controller with a scene-aware height encoder $\epsilon_{h}$: the encoder maps the local height map $h_{t}$ into terrain context for scene-conditioned whole-body control.

Sources and Demonstrations

Figures are reproduced from Xie et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

LadderMan: Learning Humanoid Perceptive Ladder Climbing
arXiv preprint

LadderMan: Learning Humanoid Perceptive Ladder Climbing

Siheng Zhao, Yuanhang Zhang, Ziqi Lu et al.

Humanoid robots hold great promise for operating in human-centered environments, yet ladder climbing remains one of the most challenging tasks due to sparse footholds and handholds, complex whole-body coordination, and sensitivity to perception and control errors.

sim-to-real reinforcement-learning vision manipulation humanoid
PDF Advanced
A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Flash-WAM: Modality-Aware Distillation for World Action Models
arXiv preprint

Flash-WAM: Modality-Aware Distillation for World Action Models

Arman Akbari, Ci Zhang, Arash Akbari et al.

World-action models (WAMs) jointly generate future video and robot actions through iterative diffusion, achieving strong performance on manipulation benchmarks but requiring tens of denoising steps, a cost that precludes real-time control.

sim-to-real reinforcement-learning diffusion-policy manipulation humanoid
PDF Intermediate