GesVLA: Gesture-Aware Vision-Language-Action Model

Wenxuan Guo, Ziyuan Li, Meng Zhang, Yichen Liu, Yimeng Dong, Chuxi Xu, Yunfei Wei, Ze Chen, Erjin Zhou, Jianjiang Feng · Tsinghua University · 2026

Summary

GesVLA augments standard VLA models with gesture awareness, enabling robots to interpret verbal instructions alongside human hand gestures for disambiguated manipulation.

Abstract Summary

GesVLA addresses a key limitation in existing Vision-Language-Action (VLA) models: their reliance on textual instructions alone, which often fails to resolve spatial ambiguity in cluttered scenes with multiple similar objects. The authors introduce a Gesture-Aware Embedding (GAE) module that fuses 2D hand pose estimates with language tokens before feeding them into the action generation head. The system architecture processes each input frame through three parallel streams: a vision encoder for scene understanding, a text tokenizer for language instructions, and a lightweight hand-pose detector for gesture cues. The GAE module aligns these modalities into a shared embedding space, allowing the model to resolve ambiguous references such as "pick the cup" by using the pointing direction of the human's hand. This multi-modal grounding produces significantly more reliable target selection in cluttered tabletop environments. GesVLA is evaluated both in high-fidelity simulation and on a physical 7-DoF Franka robotic arm. Results demonstrate improved success rates on ambiguous object-selection tasks compared to text-only VLAs, and the model generalizes to unseen object categories and novel gesture patterns. Additionally, the framework supports gesture-only commands—useful in noise-sensitive settings where verbal instructions are unreliable—making it a versatile interface for human-robot collaboration.

Key Points

First VLA model to explicitly integrate hand-gesture awareness.
Gesture-Aware Embedding (GAE) fuses 2D hand poses with language tokens.
Disambiguates object references in cluttered, multi-object scenes.
Evaluated on Franka robot with real-world and simulated tasks.
Supports gesture-only commands for noise-sensitive environments.

PROBLEM FORMULATION

We study a gesture-augmented vision-language-action (VLA) problem, where a robot is required to execute manipulation tasks based on multimodal human instructions. In standard VLA settings, the robot receives visual observations $\mathcal{O}$ and a language instruction $\mathcal{T}$, and aims to predict a sequence of actions that accomplish the task.

However, in many real-world scenarios, language alone is insufficient to uniquely specify the target due to spatial ambiguity (e.g., multiple similar objects).

PROBLEM FORMULATION — GesVLA: Gesture-Aware Vision-Language-Action Model — Fig. 2 : Overview of the proposed gesture-aware dual-VLM architecture. $\text{VLM}_{\text{int}}$ performs gesture-conditioned intent reasoning from multimodal instructions, producing textual and visual reasoning outputs. $\text{VLM}_{\text{per}}$ performs online perception by attending to the latent representations of $\text{VLM}_{\text{int}}$ via cross-attention. The action expert generates action trajectories through iterative denoising. The asymmetric attention design enables efficient inference by enforcing a unidirectional information flow, where $\text{VLM}_{\text{int}}$ does not attend to downstream modules, allowing it to be computed once and reused throughout execution.

Method

As shown in Fig. 2 , we propose a gesture-aware dual-VLM architecture that separates intent reasoning from action generation while enabling tight interaction between them. The model consists of a VLM for intent reasoning ($\text{VLM}_{\text{int}}$), a VLM for online perception ($\text{VLM}_{\text{per}}$), and an action expert ($\mathcal{F}_{\theta}$).

Experiments

The experimental setup is shown in Fig. 4 . We conduct both intent reasoning and robot manipulation experiments. Three cameras with a frame rate of 30 fps are deployed on a fixed table: two provide a global view and a right-side view, respectively, and the third is mounted on the robotic gripper. Intent Reasoning Experiment.

To validate sim-to-real transfer of gesture-conditioned intent reasoning, $\text{VLM}_{\text{int}}$ is trained on semi-synthetic data and directly evaluated in real-world scenarios.

Experiments — GesVLA: Gesture-Aware Vision-Language-Action Model — Fig. 4 : Experimental setup and tasks in real-world environments. The manipulation tasks include Pick-and-Place Block, Select Jelly, and Select Fruit/Vegetable under multimodal gesture-language instructions. We also independently evaluate gesture-conditioned intent reasoning tasks, where the model predicts target objects from pointing gestures.

Sources and Demonstrations

Figures are reproduced from Guo et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.