Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

Dechen Gao, Yue Yang, Ben Abbatematteo, Nathan Godwin, Pengcheng Wang, Roger Boldu, Steven Man, Zhiyang Dou, Chuan Qin, Sho Nakagome, Eric Whitmire · N/A · 2026

Summary

Abstract Summary

Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots. Inverse kinematics (IK) retargets human motion to robots efficiently but ignores dynamics, often producing infeasible motions. Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters. RL suffers from costly and unstable training and tedious reward engineering; MPC avoids policy optimization, yet retargets each trajectory in isolation, and solving one does not make the next easier. Sampling cost grows rapidly with dataset size and task difficulty. We hypothesize that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations, so that retargeting can be reduced to sampling from that manifold, conditioned on human motion, rather than solving a fresh optimization problem for every demonstration. We propose \textbf{Generative Neural Retargeting} (GNR), which uses a flow matching model to sample feasible trajectories. GNR outperforms MPC with only $8.5\%$ of the samples required by MPC, achieving a success rate of $56.20\%$ compared to $27.20\%$ for MPC. GNR can be used for scalable and efficient retargeting of large-scale, long-horizon, and millimeter precision human demonstrations: by applying GNR within a real-to-sim data engine, we produce a dexterous manipulation dataset with dense contact-force labels, spanning $223$k demonstrations and $3.3$k object geometries.

Key Points

Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots.
Inverse kinematics (IK) retargets human motion to robots efficiently but ignores dynamics, often producing infeasible motions.
Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters.
RL suffers from costly and unstable training and tedious reward engineering; MPC avoids policy optimization, yet retargets each trajectory in isolation, and solving one does not make the next easier.
Sampling cost grows rapidly with dataset size and task difficulty

Generative Neural Retargeting

Physics-based retargeting problem. Retargeting takes a human demonstration, represented either as human hand motion or as a robot kinematic reference obtained by IK, and asks for a dynamically feasible control sequence $\bm{u}_{0:T-1}$ over $T$ timesteps that reproduces the demonstrated behavior under physics.

We denote the reference motion as $\bar{\bm{x}}_{0:T}=(\bar{\bm{x}}_{0},\dots,\bar{\bm{x}}_{T})$, where each $\bar{\bm{x}}_{t}$ contains the robot and object states at timestep $0\leq t\leq T$.

Generative Neural Retargeting — Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation — Figure 1: We propose Generative Neural Retargeting ( GNR ), a scalable and generalizable retargeting method that retargets large-scale, long-horizon and high-precision human demonstrations for five-fingered dexterous manipulation. We apply GNR with a real-to-sim data engine that reconstructs human motion from egocentric videos or a wearable exoskeleton with motion capture to scale up both diverse and difficult high-precision tasks.

A Human-to-Robot Data Engine

GNR accelerates physics-based retargeting and drives a human-to-robot data engine ( Figure 2 ) which produces feasible robot trajectories with simulation-derived contact-force annotations across both difficulty and diversity axes. We describe the retargeting pipeline in Section 4.1 and the discovery and scaling phases for expanding dataset diversity in Section 4.2 .

The data engine takes per-frame human hand motion, object poses, and an object surface mesh $\mathcal{M}$ as input.

A Human-to-Robot Data Engine — Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation — Figure 2: The human-to-robot data engine. (a) A pipeline retargets motion-capture recordings for the difficulty axis and egocentric demonstrations for the diversity axis into candidate robot trajectories with simulation-derived contact-force labels. (b) The discovery and scaling phases repeatedly apply (a) under varied conditions and filter candidates using the same success criterion $\Gamma$ ( Section 4.2 ), first identifying retargetable segments and then expanding their physical and geometric coverage. The resulting pairs train GNR to accelerate subsequent retargeting.

Experiments

We evaluate GNR along the diversity and difficulty axes. We first describe the experimental settings ( Section 5.1 ), then evaluate GNR ’s performance and efficiency on large-scale, diverse egocentric demonstrations ( Section 5.2 ) and long-horizon, high-precision demonstrations ( Section 5.3 ). Simulator & Embodiment.

We use SuperDex Physics ( The Mochi Team, 2026 ) as the simulator, which supports non-convex collision geometry and spatially distributed 3-D contact forces, enabling dense contact-force annotations.

Experiments — Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation — Figure 3: GNR performance on HOT3D across three OOD subsets. For each subset, we report GNR ’s SRs on MPC-success, MPC-failure, and all demonstrations.

Sources and Demonstrations

Figures are reproduced from Gao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.