Robotics Research Hub
Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Tara Sadjadpour, Siming He, C. K. Wolfe, Haozhi Qi, Lea Wilken, S. Shankar Sastry, Claire Tomlin, Jitendra Malik · N/A · 2026

manipulation reinforcement-learning sim-to-real learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment.

Abstract Summary

Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand, while also improving the success rate of downstream dynamic retargeting by as much as 35 points. Ablations on the residual RL show complementary benefits from using object pose and contact information. Finally, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects. Project page: $\href{https://morphometricimitation.github.io}{\text{this https URL}}$

Key Points

  • Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment.
  • We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies.
  • First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts.
  • Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations.
  • Third, these demonstrations are distilled into visuomotor policies

Modeling Human Hands with MANO

Our method builds on the MANO hand model [ 35 ] , where the skinning function $W$ poses a rest-pose mesh $T_{P}\in\mathbb{R}^{V\times 3}$ about the rest-pose joint locations $J\in\mathbb{R}^{(K+1)\times 3}$ according to the pose $\overset{\rightarrow}{\theta}$ and the per-vertex blend weights $\mathcal{W}\in\mathbb{R}^{V\times(K+1)}$.

The rest-pose mesh and joints are obtained from a template hand mesh $\overline{\mathbf{T}}\in\mathbb{R}^{V\times 3}$ as where $\mathcal{J}\in\mathbb{R}^{(K+1)\times V}$ is a joint regressor.

Modeling Human Hands with MANO — Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Fig. 3: Morphometric Optimization. We illustrate the three stages of morphometric optimization across three diverse robot hand embodiments: Dex3, Allegro, and Sharpa. (1) Morphology matching initializes Scaled MANO using robot-to-human palm and finger-length ratios, then optimizes its morphology to match the target robot hand. (2) Contact matching optimizes the morphology-aligned hand to recover the contacts from the reference human hand-object interaction. (3) Robot pose recovery applies linear blend retargeting to transfer the resulting morphology- and contact-aligned MANO skeleton to the robot skeleton, followed by inverse kinematics to recover the robot joint configuration.

Kinematic Retargeting: Morphometric Optimization

The first stage of our pipeline converts human hand motion into robot hand motion with Morphometric Optimization (MMO) , which consists of three steps (Figure 3 ). (1) Morphology matching optimizes the MANO model to match the morphology of the robot hand.

(2) Because this morphological change can alter the hand-object contacts, contact matching re-optimizes the morphology-aligned MANO hand at each frame to recover the contacts of the original MANO hand.

Kinematic Retargeting: Morphometric Optimization — Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Fig. 4: Dynamic Retargeting Rollouts. We show time lapses of residual RL policy rollouts for three robot hand embodiments (Dex3, Allegro, and Sharpa) across three object categories (hammer, apple, and cube). All policies use the same training hyperparameters, demonstrating that the residual RL formulation transfers across distinct hand morphologies and grasp configurations without embodiment- or task-specific tuning.

Dynamic Retargeting: Residual Reinforcement Learning

Kinematic retargeting provides a reference robot trajectory $\mathbf{q}_{\text{ref}}$ that captures the desired hand motion and contact behavior but does not account for the dynamics of the robot-object interaction.

Dynamic Retargeting: Residual Reinforcement Learning — Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Fig. 5: Qualitative kinematic retargeting at the end frames of the wineglass with Sharpa (first row) and the bowl with Allegro (second row). Each row shows the human reference and the four methods with the highest F1 at the same frame for direct comparisons. Results for all methods are provided in the Supplemental Material.

Zero-Shot Sim-to-Real Visuomotor Policy

We distill each privileged residual RL teacher into a visuomotor student policy for zero-shot sim-to-real deployment. We collect demonstrations by rolling out the teacher under the domain randomizations above, except random timestep initialization, since demonstrations begin at the start of the reference trajectory.

Zero-Shot Sim-to-Real Visuomotor Policy — Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Fig. 6: Real-World Object Set. 30 objects used for zero-shot sim-to-real evaluation, comprising three instances from each of the ten object categories.

Experiments

Does MMO better preserve demonstrated hand-object contacts than prior kinematic retargeting methods across robot hand embodiments? Do the kinematic references from MMO lead to better downstream dynamic retargeting? Are both object pose and contact information important for successful dynamic retargeting?

How robustly do the resulting visuomotor policies transfer zero-shot to real-world objects with diverse geometries and physical properties across randomized poses within a $10\,\text{cm}\times 10$ cm region? We first describe the setup (Section...

Sources and Demonstrations

Figures are reproduced from Sadjadpour et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
CALM: Current Aligned Link Manipulation for Single Arm Oversized Object Lifting
arXiv preprint

CALM: Current Aligned Link Manipulation for Single Arm Oversized Object Lifting

Hu, Jun, Chen, Sihan, Jovanovic, Kosta et al.

Most robots manipulate objects solely with their end effectors, whereas humans flexibly leverage different body parts, such as the forearm and elbow, especially when handling oversized objects.

manipulation reinforcement-learning sim-to-real learning-from-demonstration
PDF Intermediate