Robotics Research Hub
PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

Xiaomeng Xu, Han Zhang, Zhian Ruan, Xiaofeng Guo, Jinkun Cao, Ruihan Yang, Carmelo Sferrazza, Shuran Song, Guanya Shi · N/A · 2026

manipulation vision reinforcement-learning control learning-from-demonstration
PDF arXiv

Framework

N/A

License

N/A

Stars

N/A

Summary

Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment.

Abstract Summary

Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io

Key Points

  • Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment.
  • To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations.
  • Our key idea is to use contact as manipulation-centric information that transfers across embodiments
  • Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints.
  • Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces.

PanTact Data Collection Interface

PanTact’s data collection interface combines egocentric-augmented bimanual UMIs ( Chi et al., 2024 ; Xu et al., 2026 ) with modular contact patches (Fig. 2 ), capturing vision and motion together with body-contact signals that identify where and when contact occurs.

PanTact Data Collection Interface — PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Figure 1: PanTact gives a humanoid whole-body dexterity, learned from robot-free human demonstrations. (a) The data collection interface captures whole-body contacts alongside vision and motion. (b) Contact provides an interface for human-to-robot transfer. The visuomotor policy predicts contact goals, and the whole-body controller realizes them using suitable robot surfaces. (c) PanTact enables diverse whole-body dexterous skills on a humanoid.

Contact-Predictive Whole-Body Manipulation Policy

To learn whole-body dexterity from human demonstrations, we employ a policy that predicts whole-body contact goals alongside EEF trajectories (Fig. 3 ). To mitigate the human-to-robot embodiment gap, the policy masks body regions from the egocentric view (§ 4.1 ) and predicts world-frame EEF and contact goals as actions (§ 4.2 ). Whole-body contacts often occur behind the grippers, outside the views of wrist cameras.

Contact-Predictive Whole-Body Manipulation Policy — PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Figure 2: PanTact modular contact patch. Sensor, connector, and strap modules snap together into patches of arbitrary shape and extent. We showcase arm and knee patches.

Contact-Aware Whole-Body Control

The whole-body controller converts the policy’s EEF trajectories and contact goals $\{(\hat{c}_{i},\hat{\mathbf{p}}_{i})\}$ into robot whole-body motions. It tracks contact locations only when $\hat{c}_{i}=1$, prioritizing task-critical interactions without requiring the robot to reproduce demonstrator’s free-space body motion.

Each active goal can be realized by any suitable robot surface, leaving the controller to coordinate the remaining degrees of freedom under robot-specific constraints. Any-surface contact objective.

Contact-Aware Whole-Body Control — PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Figure 3: Cross-Embodiment Contact Interface. The Contact-Predictive Whole-Body Manipulation Policy uses body-masked 3D egocentric observations, wrist images, and proprioception to predict EEF trajectories and contact goals (position and binary state). The Contact-Aware Whole-Body Controller coordinates whole-body motion to track EEF commands and realize active contact goals using suitable robot surfaces.

Evaluation

We evaluate whether whole-body dexterity can be learned from robot-free human demonstrations and transferred to a humanoid. Our evaluation probes four capabilities: Cross-embodiment contact transfer: contacts measured on a human body and predicted by the visuomotor policy must be realized by a humanoid with a different configuration. This capability is required for all tasks.

Non-prehensile upper-body manipulation: supporting bulky objects with the arms (§ 6.1 , 6.3 ) and sweeping objects with the arm surfaces (§ 6.2 ).

Evaluation — PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface
Figure 4: Grasp-and-Embrace. (a) Autonomous rollout of our policy deployed with Upper-Body IK , demonstrating capability to support a bulky object with arms while grasping small object with a gripper. (b) Overlayed test configurations with varied object placements and initial robot configurations. (c) Typical failure case of Humanoid UMI is insufficient arm contact to support the bulky object. (d) Task success rates. (e) Real-world tracking plots.

Sources and Demonstrations

Figures are reproduced from Xu et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced