GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

Zhihai Bi, Qiang Zhang, Guoyang Zhao, Jiahang Cao, Xueyin Luo, Yushan Zhang, Jinglan Xu, Ruoyu Geng, Yulin Li, Andrew F. Luo, Jun Ma · N/A · 2026

Summary

Abstract Summary

Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present GenHOI, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.

Key Points

GenHOI enables zero-shot humanoid-object interaction by imitating a single generated video.
Reconstructs robot-object scenes in simulation to condition video generation with language commands.
Extracts contact events and geometric constraints from generated videos for physically grounded motion optimization.
Validates on diverse real-world tasks including bimanual chair carrying and table lifting.

Methodology

In this work, we propose GenHOI, a humanoid loco-manipulation framework that directly imitates generated videos without task-specific training, as illustrated in Fig. 2 . Specifically, the proposed framework comprises four key stages: real-to-sim video generation (Sec. III-A ), contact-aware geometric constraint extraction (Sec. III-B ), geometry-guided trajectory optimization (Sec.

Experiments

This section presents the experimental results to systematically evaluate the proposed method. Specifically, we organize our experiments to answer the following questions: $\mathcal{Q}_{1}$: How does GenHOI perform compared to other methods? (Sec. IV-B ) $\mathcal{Q}_{2}$: What key factors influence the performance of video generation and downstream execution for HOI? (Sec.

IV-C ) $\mathcal{Q}_{3}$: How robust are the key modules, including the VLM-based key frame selection and hand-object contact estimation? (Sec.

Experiments — GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training — Fig. 3: Qualitative comparison of different methods on loco-manipulation tasks across four object categories. Red crosses denote failed trials, and green check marks denote successful task completion.

Sources and Demonstrations

Figures are reproduced from Bi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.