AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction

Seok Joon Kim, Junho Lee, Federica Spinola, Taein Kwon, Mohsen Moghaddam · N/A · 2026

Summary

Abstract Summary

Direct hand-driven teleoperation maps an operator's hand motion to robot end-effector commands at every frame, enabling precise control, but it requires constant monitoring and correction during approach, grasp, and placement, which can be slow and fatiguing. For repetitive pick-and-place tasks, supervisory (goal-based) teleoperation simplifies this process: the operator specifies goals/waypoints, and the robot executes the motion using planning algorithms. Yet, this introduces latency, as the robot must wait for the next command before it can plan and act. "How can we reduce robot reaction time while lowering operator workload?" To tackle this question, we present AHEAD, a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control. In a digital twin, the operator performs pick-and-place naturally, using hand motion to convey high-level commands rather than a continuous robot trajectory. AHEAD processes a short window of 3D hand and head signals together with scene context through an attention-based classifier to predict the intended grasp object and placement slot. A state machine converts intent predictions into stable robot goals, enabling early motion while remaining stable under noisy predictions and corrective hand movements. AHEAD's intent prediction module achieves Top1 accuracy: 76% for grasp objects and 76% for target slots. Moreover, our user study shows AHEAD reduces robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines. Participants also reported lower operator load, indicating faster robot responses while maintaining low operator effort in practice.

Key Points

Direct hand-driven teleoperation maps an operator's hand motion to robot end-effector commands at every frame, enabling precise control, but it requires constant monitoring and correction during approach, grasp, and placement, which can be slow and fatiguing.
For repetitive pick-and-place tasks, supervisory (goal-based) teleoperation simplifies this process: the operator specifies goals/waypoints, and the robot executes the motion using planning algorithms.
Yet, this introduces latency, as the robot must wait for the next command before it can plan and act
"How can we reduce robot reaction time while lowering operator workload?" To tackle this question, we present AHEAD, a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control.
In a digital twin, the operator performs pick-and-place naturally, using hand motion to convey high-level commands rather than a continuous robot trajectory.

Method

This section presents AHEAD , a VR teleoperation system deployed in a digital twin workspace. It predicts operator intent to proactively initiate robot motion and reduce robot reaction time , shown as star markers in Fig. 1 . We consider a pick-and-place teleoperation task in which the robot operates in a physical workspace while the operator interacts remotely through a mirrored VR digital twin (see top row of Fig.

1 ). Using a Meta Quest 3, the operator views the digital twin and controls the robot through natural reach-and-place motions.

Method — AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction — Fig. 2: Data collection procedure: A left-hand pinch starts recording (2), then the right hand reaches (3) and grasps (4) the object, and transports (5) and places it in the slot (6). After release (7), operators can redo (left-hand pinch) or advance (left-hand grasp) (8). Yellow boxes highlight key events.

Evaluation

We evaluate AHEAD in two complementary settings: (i) offline intent prediction on the collected dataset, and (ii) an user study that measures the benefits of proactive assistance during teleoperation.

We evaluate offline intent prediction using a 5-fold leave-one-subject-out (LOSO) protocol: in each fold, we train on data from four participants and test on the held-out participant (200 sequences per participant, 1,000 sequences total).

Evaluation — AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction — Fig. 3: Phase-specific intent prediction model architecture. Depending on the phase, we fuse hand-head features (A) with either object (B) or slot (C) embeddings via cross-attention (D) and predict the target (E).

Sources and Demonstrations

Figures are reproduced from Kim et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.