Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Jusuk Lee, Sungha Kim, Yeonsoo Park, Jonguk Cheon, Yoonkyo Jung, Yongjun You, H. Jin Kim, Jia-Bin Huang, Furong Huang, Youngseok Jang, Seungjae Lee · N/A · 2026

Summary

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions.

Abstract Summary

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.

Key Points

While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions.
Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video.
Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks.
To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video.
Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability.

Method

Given a single demonstration of a manipulation task, our goal is to learn a dexterous policy that generalizes to diverse spatial configurations, including randomized object poses and novel grasps, beyond the original video. To achieve this, we first abstract the demonstration into a stage-wise scene graph (Sec. 3.1 ).

We then formulate policy learning as a Markov decision process (MDP) grounded in this graph, with observations, rewards, and an initial-state distribution constructed from it without task-specific design (Sec. 3.2 ).

Method — Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration — Figure 1 : From one human video to many dexterous behaviors. Given one human video of a manipulation task, Dex-One2Many learns a dexterous policy entirely in simulation and deploys it zero-shot in the real world. By training across diverse simulated task configurations, the resulting policy executes the task across many unseen configurations beyond the execution shown in the video.

Experiments

In this section, we evaluate Dex-One2Many in simulation and on real hardware. Our experiments address three questions: Does Dex-One2Many generalize to initial and goal configurations not shown in the single human video? Does Dex-One2Many apply across diverse dexterous embodiments with different kinematics? Which design choices of Dex-One2Many are critical to its performance? Tasks.

We evaluate Dex-One2Many on five manipulation tasks that span direct object manipulation and tool use (Fig. 4 ).

Experiments — Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration — Figure 2 : Overview of Dex-One2Many. (1) From a single human video, a VLM extracts a stage-wise scene graph of object relations. (2) Each stage graph constrains the sampling of object poses and synthesized grasps, yielding verified reset states for every stage. An RL policy trained from these resets with predicate-derived rewards acts in task space through a QP controller. (3) The policy transfers zero-shot to unseen initial poses, goal poses, and grasps on real hardware.

Sources and Demonstrations

Figures are reproduced from Lee et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.