OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video

Ting Mao, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu, Yao Mu · N/A · 2026

Summary

Abstract Summary

Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39-89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.

Key Points

Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution.
Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video.
We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands.
The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement.
Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy.

Method

As shown in Figure 2 , OmniHOI samples keyframes from the video according to hand motion, reconstructs and refines the hand-object interaction in each keyframe ( Section 3.1 ), retargets the refined MANO hands ( Romero et al., 2017 ) to the robot hand under contact guidance, and turns the keyframes into a continuous, task-effective trajectory in simulation ( Section 3.2 ).

We use SAM 3 ( Carion et al., 2026 ) to segment hands, objects and table, and use MoGe-3 to estimate camera intrinsics and each frame’s depth as point map.

Method — OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video — Figure 1: From human video to robot execution. Top: monocular RGB videos of human hand-object interaction. Bottom: the trajectories OmniHOI produces from them, executed on a pair of XHands.

Experiments

We follow the setup and criteria from prior work, testing our method on three public datasets: ARCTIC ( Fan et al., 2023 ) , DexYCB ( Chao et al., 2021 ) and OakInk2 ( Zhan et al., 2024 ) , all with 3D hand-object pose and shape annotations.

ARCTIC includes humans operating 11 articulated objects, OakInk2 features manipulation tasks with 75 diverse objects, and DexYCB presents grasping of 20 distinct YCB-Video ( Xiang et al., 2018 ) objects.

Experiments — OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video — Figure 2: OmniHOI is built on several key designs. Adaptive sampling reduces the video clip to keyframes (left) . Hand pose in each keyframe is reconstructed, refined and retargeted with contact information preserved (middle) . Interpolating between the keyframe configurations then gives an initial robot trajectory. Physics-in-the-loop refinement then rolls the trajectory out in simulation and updates the keyframe configurations with CMA-ES, yielding a task-effective trajectory (right) .

Sources and Demonstrations

Figures are reproduced from Mao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.