VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity

Problem Formulation

We consider a dexterous hand manipulating rigid objects under partial observability. At each control step $t$, the policy observes proprioception $q_{t}$, an optional point cloud $p_{t}$, and a tactile representation $z_{t}$. It outputs a continuous action $a_{t}$ for the arm and hand joints.

The goal is to maximize task return for contact-rich manipulation tasks such as grasping, lifting, placing, inserting, and in-hand repositioning. The tactile hardware consists of piezoelectric microphones embedded in each robot fingertip (Fig. 1 , left).

Method

VibeAct has three components (Fig. 1 ). First, a teleoperation pipeline records robot and object states, synchronized with vibro-acoustic signals, then replays trajectories in a calibrated digital clone in simulation to generate per-finger contact and slip labels as a tactile representation. Second, a tactile estimator learns to map real microphone signals to this tactile representation.

Third, RL policies are trained in simulation using proprioception, point clouds, and the same tactile representation as observations.

Experiments

We evaluate VibeAct along three axes. We assess how accurately the tactile estimator recovers contact and slip from real microphone signals (Sec. 5.1 ), whether the resulting representation improves manipulation policies in simulation and which channels drive the gains (Sec. 5.3 ), and whether the benefit transfers to a physical robot.

Together they test whether a physically grounded tactile representation can bridge real vibrotactile sensing and simulation-based policy learning.

Sources and Demonstrations

Figures are reproduced from Mao et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.