Robotics Research Hub
SEED-UMI: Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration

SEED-UMI: Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration

Tengbo Yu, Jiahao Wu, Daohan Li, Bingxu Chen, Hao Liu, Xiaojian Ma, Hangxin Liu · N/A · 2026

manipulation vision reinforcement-learning learning-from-demonstration
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Imitation learning for dexterous hands is bottlenecked by the difficulty of collecting contact-rich demonstrations that transfer faithfully to the robot.

Abstract Summary

Imitation learning for dexterous hands is bottlenecked by the difficulty of collecting contact-rich demonstrations that transfer faithfully to the robot. Prior wearable-exoskeleton systems record only on the human side and retarget via open-loop mappings calibrated in free space, which degrade under contact. We present SEED-UMI, a framework in which both the human and the robot wear the same exoskeleton: joint encoders become a physically shared measurement, and wrist cameras mounted to the exoskeleton observe the same outer mechanism during both human data collection and robot policy rollouts. This turns retargeting into paired cross-embodiment supervision and lets policies train directly on raw exoskeleton-centric wrist images, without segmentation or inpainting. On five contact-rich tasks, SEED-UMI achieves 3.0x greater data collection efficiency than exoskeleton-based teleoperation and a 70.0% average rollout success rate.

Key Points

  • Imitation learning for dexterous hands is bottlenecked by the difficulty of collecting contact-rich demonstrations that transfer faithfully to the robot.
  • Prior wearable-exoskeleton systems record only on the human side and retarget via open-loop mappings calibrated in free space, which degrade under contact.
  • We present SEED-UMI, a framework in which both the human and the robot wear the same exoskeleton: joint encoders become a physically shared measurement, and wrist cameras mounted to the exoskeleton observe the same outer mechanism during both human data collection and robot policy rollouts.
  • This turns retargeting into paired cross-embodiment supervision and lets policies train directly on raw exoskeleton-centric wrist images, without segmentation or inpainting.
  • On five contact-rich tasks, SEED-UMI achieves 3.0x greater data collection efficiency than exoskeleton-based teleoperation and a 70.0% average rollout success rate.

Hardware: A Shared Exoskeleton

The hardware of SEED-UMI matches the contact-facing exoskeleton geometry and encoder frame on the human and robot sides, while adapting cuffs and mounts to each embodiment. This makes encoder readings a shared cross-embodiment measurement : discrepancies under the same command supervise the residual command-to-joint mapping (§ 4 ). Figure 2 shows both systems.

The exoskeleton is co-designed with a single fully-actuated dexterous hand as its sole target platform.

Hardware: A Shared Exoskeleton — SEED-UMI: Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration
Figure 3: SEED-UMI pipeline. Human demonstrations are collected through the shared exoskeleton, paired robot replay refines the encoder-to-command mapping, and the learned policy rolls out on the robot with matched exoskeleton-centric observations.

Software Adaptation: Two-Stage Cross-Embodiment Mapping

The shared exoskeleton gives both embodiments a common encoder signal $\mathbf{e}_{t}\in\mathbb{R}^{20}$, but robot execution still requires an encoder-to-command mapping $\mathbf{e}_{t}\mapsto\mathbf{q}_{t}$. Instead of fitting this mapping once in free space [ 34 ] , we bootstrap it with robot motor babbling and refine it with paired replay under real contact.

The mapping is proprioceptive, while aligned wrist images are reserved for policy learning; Figure 3 shows the full pipeline.

Software Adaptation: Two-Stage Cross-Embodiment Mapping — SEED-UMI: Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration
Figure 4: Hardware Design. (a) shows the collision avoidance of our design during operation. (b) shows how the linkage work during operation.

Evaluation

Our platform consists of a RealMan RX75 robot arm and a Wuji dexterous hand with four actuated DoFs per finger (20 in total). The human and robot wear the shared exoskeleton described in § 3 , preserving the outer contact geometry and camera placement across embodiments.

A dorsal Intel RealSense T265 provides 6-DoF wrist pose, while a ventral fisheye camera with a $150^{\circ}$ field of view supplies policy observations. Wrist trajectories use a fixed T265-to-tool-center-point (TCP) alignment without trajectory correction.

Evaluation — SEED-UMI: Sharing the Exoskeleton between human and robot for onE-to-one Dexterous demonstration
Figure 5: Policy Rollouts. We evaluate SEED-UMI on five contact-rich dexterous manipulation tasks that together stress tool stabilization, index–middle finger coordination, grasp-to-release timing, contact-rich table wiping, and thumb-involved multi-finger actuation.

Sources and Demonstrations

Figures are reproduced from Yu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced