GesVLA: Gesture-Aware Vision-Language-Action Model

GesVLA: Gesture-Aware Vision-Language-Action Model

Featured

Wenxuan Guo, Ziyuan Li, Meng Zhang, Yichen Liu, Yimeng Dong, Chuxi Xu, Yunfei Wei, Ze Chen, Erjin Zhou, Jianjiang Feng · Tsinghua University · 2026

Framework

N/A

License

N/A

Stars

18

Summary

GesVLA augments standard VLA models with gesture awareness, enabling robots to interpret verbal instructions alongside human hand gestures for disambiguated manipulation.

Abstract Summary

GesVLA addresses a key limitation in existing Vision-Language-Action (VLA) models: their reliance on textual instructions alone, which often fails to resolve spatial ambiguity in cluttered scenes with multiple similar objects. The authors introduce a Gesture-Aware Embedding (GAE) module that fuses 2D hand pose estimates with language tokens before feeding them into the action generation head. The system architecture processes each input frame through three parallel streams: a vision encoder for scene understanding, a text tokenizer for language instructions, and a lightweight hand-pose detector for gesture cues. The GAE module aligns these modalities into a shared embedding space, allowing the model to resolve ambiguous references such as "pick the cup" by using the pointing direction of the human's hand. This multi-modal grounding produces significantly more reliable target selection in cluttered tabletop environments. GesVLA is evaluated both in high-fidelity simulation and on a physical 7-DoF Franka robotic arm. Results demonstrate improved success rates on ambiguous object-selection tasks compared to text-only VLAs, and the model generalizes to unseen object categories and novel gesture patterns. Additionally, the framework supports gesture-only commands—useful in noise-sensitive settings where verbal instructions are unreliable—making it a versatile interface for human-robot collaboration.

Key Points

  • First VLA model to explicitly integrate hand-gesture awareness.
  • Gesture-Aware Embedding (GAE) fuses 2D hand poses with language tokens.
  • Disambiguates object references in cluttered, multi-object scenes.
  • Evaluated on Franka robot with real-world and simulated tasks.
  • Supports gesture-only commands for noise-sensitive environments.

Additional Notes

Overview

  • OpenVLA: An Open-Source Vision-Language-Action Model
  • RT-2: Vision-Language-Action Models
  • RT-1: Robotics Transformer for Real-World Control at Scale
Share

Related Papers

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

3DThinkVLA: Endowing Vision-Language-Action Models with Latent 3D Priors via 3D-Thinking-Guided Co-training

Jiaxin Shi, Xidong Zhang, Fucai Zhu et al. · arXiv preprint · Jun 2026

We propose a 3D-thinking-guided co-training framework that enables vision-language-action (VLA) models to perform 3D spatial reasoning implicitly during action prediction. Our core insight is that 3D geometry perception and 3D spatial reasoning are distinct capabilities that can be disentangled and ...

sim-to-real reinforcement-learning vision vla manipulation
PDF Advanced
No code repo Jun 2026
AIR-VLA+: Decoupling Movement and Manipulation via Cascaded Dual-Action Decoders with Asymmetric MoE for Aerial Robots

AIR-VLA+: Decoupling Movement and Manipulation via Cascaded Dual-Action Decoders with Asymmetric MoE for Aerial Robots

Jianli Sun, Bin Tian, Qiyao Zhang et al. · arXiv preprint · Jun 2026

Aerial manipulation systems have long suffered from representation coupling in end-to-end control, as platform-level Unmanned Aerial Vehicle (UAV) movement and end-effector-level arm manipulation differ substantially in action scale, dynamics, and control objectives. In this paper, we propose AIR-VL...

uav vla manipulation
PDF Advanced
No code repo Jun 2026
AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation

AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation

Mengfei Zhao, Dihong Huang, Yikai Tang et al. · arXiv preprint · Jul 2026

Learning effective robot manipulation policies requires diverse, high-quality demonstrations, yet existing data pipelines are often difficult to scale because they rely on specialized hardware, centralized operators, or fixed task suites. We present AXIS, a growable community-driven data engine a...

manipulation vision vla learning-from-demonstration benchmark
PDF Advanced
No code repo Jul 2026