Robotics Research Hub
WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control

WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control

Haoran Jiang, Jin Chen, Qingwen Bu, Li Chen, Modi Shi, Yanjie Zhang, Delong Li, Chuanzhe Suo, Chuang Wang, Zhihui Peng, Hongyang Li · N/A · 2025

manipulation locomotion vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Humanoid robots require precise locomotion and dexterous manipulation to perform challenging loco-manipulation tasks. Yet existing approaches, modular or end-to-end, are deficient in manipulation-aware locomotion.

Abstract Summary

Humanoid robots require precise locomotion and dexterous manipulation to perform challenging loco-manipulation tasks. Yet existing approaches, modular or end-to-end, are deficient in manipulation-aware locomotion. This confines the robot to a limited workspace, preventing it from performing large-space loco-manipulation. We attribute this to: (1) the challenge of acquiring loco-manipulation knowledge due to the scarcity of humanoid teleoperation data, and (2) the difficulty of faithfully and reliably executing locomotion commands, stemming from the limited precision and stability of existing RL controllers. To acquire richer loco-manipulation knowledge, we propose a unified latent learning framework that enables Vision-Language-Action (VLA) system to learn from low-cost action-free egocentric videos. Moreover, an efficient human data collection pipeline is devised to augment the dataset and scale the benefits. To execute the desired locomotion commands more precisely, we present a loco-manipulation-oriented (LMO) RL policy specifically tailored for accurate and stable core loco-manipulation movements, such as advancing, turning, and squatting. Building on these components, we introduce WholeBodyVLA, a unified framework for humanoid loco-manipulation. To the best of our knowledge, WholeBodyVLA is one of its kind enabling large-space humanoid loco-manipulation. It is verified via comprehensive experiments on the AgiBot X2 humanoid, outperforming prior baseline by 21.3%. It also demonstrates strong generalization and high extensibility across a broad range of tasks.

Key Points

  • Humanoid robots require precise locomotion and dexterous manipulation to perform challenging loco-manipulation tasks.
  • Yet existing approaches, modular or end-to-end, are deficient in manipulation-aware locomotion
  • This confines the robot to a limited workspace, preventing it from performing large-space loco-manipulation.
  • We attribute this to: (1) the challenge of acquiring loco-manipulation knowledge due to the scarcity of humanoid teleoperation data, and (2) the difficulty of faithfully and reliably executing locomotion commands, stemming from the limited precision and stability of existing RL controllers.
  • To acquire richer loco-manipulation knowledge, we propose a unified latent learning framework that enables Vision-Language-Action (VLA) system to learn from low-cost action-free egocentric videos.

WholeBodyVLA

WholeBodyVLA, which equips VLA models with locomotion primitives that reliably establish the preconditions for manipulation, leverages unified latent learning and a loco–manipulation–oriented (LMO) RL policy to enable humanoids to accomplish long-range, large-area tasks.

In this section, we detail (i) how unified latent learning contributes to WholeBodyVLA jointly master manipulation and locomotion primitives, and (ii) how our loco–manipulation–oriented RL policy explicitly optimizes stability under dynamic disturbances.

WholeBodyVLA — WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control
Figure 2: Pipeline of WholeBodyVLA. LAM is pretrained on manipulation and manipulation-aware locomotion videos, yielding unified latent supervision for the VLM. Meanwhile, the LMO RL policy is trained for precise and stable locomotion under disturbances. At runtime, egocentric images and language instructions are encoded by the VLM into latent action tokens, which are decoded ($\sim$ 10 Hz) into (i) dual-arm joint actions and (ii) locomotion commands executed by LMO at 50 Hz, enabling robust whole-body loco–manipulation.

Experiments

Q1. (Section 4.2 ) Does WholeBodyVLA enable long-range, large-area loco–manipulation beyond existing SOTA approaches? Q2. (Section 4.3 ) Does learning from action-free videos actually improve performance and reduce reliance on teleoperation data? Q3. (Section 4.4 ) How does LMO contribute to loco–manipulation? Q4. (Section 4.5 ) Does WholeBodyVLA generalize to long-horizon and extended scenarios ?

Experiments — WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control
Figure 3: Real-world generalization of WholeBodyVLA. Top: variations in robot start-pose and scene appearance, with data-scaling curves. Bottom: comparison on extended tasks with different baselines. See videos on https://opendrivelab.com/WholeBodyVLA .

Sources and Demonstrations

Figures are reproduced from Jiang et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced
Capability-Aware Arbitration for Semantic Intent-Based Shared Control
arXiv preprint

Capability-Aware Arbitration for Semantic Intent-Based Shared Control

Zhaoda Du, Michael Bowman, Xiaoli Zhang

Shared control often allocates robot authority based on confidence in inferred human intent, assuming reliable autonomous execution. When this assumption fails, high intent confidence can cause over-helping.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Intermediate
GitHub ★ —
DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation
arXiv preprint

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

Yunchao Yao, Zhuxiu Xu, Tianqi Zhang et al.

Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments.

manipulation vision vla reinforcement-learning control learning-from-demonstration benchmark
PDF Advanced