GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan · NVIDIA, UCLA · 2026

Summary

Abstract Summary

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84% real-world success on diverse object pick-up and 90% success on stair-climbing.

Key Points

Demonstrates humanoid robot control on real hardware
Addresses dexterous manipulation challenges
Uses simulation or synthetic data for training
Leverages 3D geometric reasoning for perception
Enables complex whole-body climbing behaviors
Learns multimodal motion skills from human demonstrations

GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors

Authors: Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan

Venue: arXiv preprint | Year: 2026

arXiv: 2606.05160v1

Abstract

Key Contributions

Demonstrates humanoid robot control on real hardware
Addresses dexterous manipulation challenges
Uses simulation or synthetic data for training
Leverages 3D geometric reasoning for perception
Enables complex whole-body climbing behaviors
Learns multimodal motion skills from human demonstrations

Topics

sim-to-real
reinforcement-learning
vision
manipulation
humanoid

Code & Data

|| Resource | Link | |---|---| | Project Page | https://research.nvidia.com/labs/dair/grail/ | | GitHub | NVlabs/GRAIL (⭐ 275, Python) | | Dataset | HuggingFace |

BibTeX

@article{Xie2026_260605160v1,
  title={GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors},
  author={Tianyi Xie and Haotian Zhang and Jinhyung Park and Zi Wang and Bowen Wen and Jiefeng Li and Xueting Li and Qingwei Ben and Haoyang Weng and Yufei Ye and David Minor and Tingwu Wang and Chenfanfu Jiang and Sanja Fidler and Jan Kautz and Linxi Fan and Yuke Zhu and Zhengyi Luo and Umar Iqbal and Ye Yuan},
  year={2026},
  eprint={2606.05160v1},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2606.05160v1}
}