ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback

Zimo He, Tianyi Bian, Zizhong Yang, Hongjie Li, Nan Jiang, Jieming Cui, Baoxiong Jia, Yizhou Wang, Siyuan Huang, Yixin Zhu · N/A · 2026

Summary

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors.

Abstract Summary

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers.

Key Points

Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors.
Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback.
We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm
ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization.
We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency.

ViT-H

Our goal is to convert the human motion in a monocular video $\mathbf{I}=\{\mathbf{I}_{t}\}_{t=1}^{T}$ into whole-body actions that make a Unitree G1 humanoid reproduce it. We decompose this problem into reference motion generation and whole-body tracking.

A video-conditioned generative model $\pi_{\text{gen}}$ predicts a reference humanoid motion $\mathbf{X}\sim\pi_{\text{gen}}(\cdot\mid\mathbf{I})$, and a whole-body tracking policy $\pi_{\text{track}}$ executes it.

ViT-H — ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback — Figure 2: Overview of ViT-H . The diffusion-based generator (left) maps the camera orientations, bounding boxes, keypoints, and image features of a video directly to G1 reference motions, and its autoregressive variant also conditions on the preceding segment. For post-training (middle), candidates sampled for the same video are executed by a deployable tracker, which scores local pose accuracy and success, and by a privileged tracker, which scores global motion accuracy and success. At deployment (right), the offline generator drives video-to-humanoid control and the autoregressive generator drives real-time teleoperation.

Experiments

In this section, we first compare ViT-H with reconstruction-retargeting-tracking pipelines and our pre-training-only ablation variants on both datasets with 3D ground-truth annotations and video-only datasets ( Section 4.1 ). We then provide an empirical analysis of DPO post-training, investigating how preference construction and post-training design choices affect humanoid motion executability ( Section 4.2 ).

Sources and Demonstrations

Figures are reproduced from He et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.