ContactMimic: Humanoid Object Interaction via Contact Control

Summary

Keypoint tracking alone is insufficient for object interaction tasks such as sitting on a chair, wiping a board, or pushing furniture, where the robot can reach the correct pose without making meaningful physical contact with the object.

Abstract Summary

Key Points

• Keypoint tracking alone is insufficient for object interaction tasks such as sitting on a chair, wiping a board, or pushing furniture, where the robot can reach the correct pose without making meaningful physical contact with the object.
• We present CONTACTMIMIC, a learning framework that tracks explicit partlevel binary contact commands alongside keypoint trajectories.
• CONTACTMIMIC is made possible through the use of contact-following rewards and a trajectory augmentation scheme aimed at breaking the correlations between keypoint trajectories and contact labels.
• The resulting policy successfully decouples contact behavior from keypoint geometry, and achieves precise physical contact as well as contact-controllability (produce or suppress contact during deployment as desired).
• Simulation experiments across 10 diverse human-object interaction motions confirm that CONTACTMIMIC exhibits contact controllability that enables it to complete manipulation tasks without task-specific rewards, while also outperforming keypoint-only trackers on contact-relevant tasks.

Method

Our proposed contact and keypoint tracker, $\pi_{\theta}(\mathbf{a}_{t}/\mathbf{p}_{t},\bar{\mathbf{k}}_{t},\bar{\mathbf{c}}_{t})$, takes as input proprioception $\mathbf{p}_{t}$, reference keypoint position targets $\bar{\mathbf{k}}_{t}$, and a binary reference contact label map $\bar{\mathbf{c}}_{t}\in\{0,1\}^{/\mathcal{B}/}$, where $\mathcal{B}$ is the set of contact-capable robot bodies (pelvis, torso, hips, knees, ankles, shoulders, and wrists).

The policy outputs target joint angles that are converted into torques using a PD controller.

Experiments

Our experiments across diverse human-object interaction motions seek to answer 5 key questions: Do our policies follow the commanded contact conditioning? Does contact control transfer to the real world? Is keypoint control alone sufficient, or does a state-of-the-art keypoint-tracking baseline already produce accurate physical contact? Is our data engineering strategy necessary for learning contact control?

To what extent does the policy’s internal representation already encode the robot’s actual runtime contact state, given only proprioceptiv...

Sources and Demonstrations

Figures are reproduced from Li et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.