Robotics Research Hub
VioLA: Learning Generalist Humanoid Control Policies from Human Data

VioLA: Learning Generalist Humanoid Control Policies from Human Data

Mert Albaba, Jens Beißwenger, Anna Manasyan, Daniel Marta, Michael J. Black, Wieland Brendel, Andreas Krause, Georg Martius, Martin Riedmiller · N/A · 2026

manipulation locomotion vla reinforcement-learning control learning-from-demonstration
PDF arXiv Code (GitHub)

Framework

N/A

License

N/A

Stars

N/A

Summary

Teaching a humanoid to follow instructions with its whole body runs into two obstacles. Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn.

Abstract Summary

Teaching a humanoid to follow instructions with its whole body runs into two obstacles. Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn. And humanoid demonstrations are scarce, so current humanoid generalist policies do not follow new instructions out of the box and are fine-tuned on teleoperated demonstrations of each task before deployment. Human demonstrations exist in far larger numbers, but a person's motion is not a robot command. We remove both obstacles by changing what the generalist policy predicts. We introduce VioLA, a generalist humanoid policy that predicts body and hand motion latents instead of joint commands. A pretrained body- and hand-controller execute these latents on the robot. Their corresponding motion encoders map human and robot motion into the same latent spaces. A human recording is therefore labeled in the policy's action space, and the training demonstration pool contains 140.6 million frames, 93.2% of them human. As a result, VioLA follows locomotion instructions on the real robot zero-shot, without task-specific fine-tuning, reaching 100% success where GR00T N1.7 and $Ψ_0$ reach 16.7% and 0%, respectively. It also reaches 88.6% manipulation success without task-specific fine-tuning. The same approach works across two VLA and one world-action model backbones. A generalist policy trained on human demonstrations alone performs locomotion tasks on the real robot zero-shot. Code and checkpoints will be released.

Key Points

  • Teaching a humanoid to follow instructions with its whole body runs into two obstacles
  • Its action space is large and tightly coupled: legs, arms, and fingers must move together while the robot keeps its balance, which makes joint-level actions hard to learn.
  • And humanoid demonstrations are scarce, so current humanoid generalist policies do not follow new instructions out of the box and are fine-tuned on teleoperated demonstrations of each task before deployment.
  • Human demonstrations exist in far larger numbers, but a person's motion is not a robot command
  • We remove both obstacles by changing what the generalist policy predicts

VioLA

VioLA learns which body and hand motions an instruction requires while pretrained decoders produce the robot’s joint commands. The key choice is to use the shared latent spaces of paired motion encoders and decoders as VioLA’s action space (Figure 2 ). Human and robot encoders map demonstrated movements into this space, supplying the targets the generalist policy learns to predict.

All motion encoders and decoders are pretrained and remain frozen; training updates only the generalist policy.

VioLA — VioLA: Learning Generalist Humanoid Control Policies from Human Data
Figure 1: VioLA. Human and robot motion encoders produce targets in shared body and hand latent spaces. The generalist policy predicts motion latents from vision, language, and proprioception; frozen robot controllers execute its predictions. Annotated human motion therefore supervises the policy in the same action representation as robot motion.

Experiments

Can VioLA follow locomotion and manipulation instructions on a real humanoid without task-specific fine-tuning? We first evaluate task completion, then examine the motion predictions behind these behaviors. The remaining experiments test what human demonstrations contribute, which backbone learns humanoid behavior most effectively, and whether successive predictions produce continuous motion.

We evaluate on a Unitree G1 with five-fingered Inspire hands in our laboratory.

Experiments — VioLA: Learning Generalist Humanoid Control Policies from Human Data
Figure 2: VioLA overview. Left: frozen motion encoders construct body and hand action targets from human or robot demonstrations. Center: the generalist policy predicts motion latents from an egocentric image, an instruction, and proprioception. It is the only component updated during training. Right: a frozen body controller and hand decoder convert the predicted latents into robot joint commands. Human and robot encoders produce targets for these same decoders.

Sources and Demonstrations

Figures are reproduced from Albaba et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers

WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control
arXiv preprint

WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control

Haoran Jiang, Jin Chen, Qingwen Bu et al.

Humanoid robots require precise locomotion and dexterous manipulation to perform challenging loco-manipulation tasks. Yet existing approaches, modular or end-to-end, are deficient in manipulation-aware locomotion.

manipulation locomotion vision vla reinforcement-learning control learning-from-demonstration benchmark
Code PDF Advanced
GitHub ★ —
A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies
arXiv preprint

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

Mathilde Kappel, Clémence Grislain, Mohamed Chetouani et al.

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions.

manipulation vision vla reinforcement-learning sim-to-real control learning-from-demonstration benchmark
PDF Advanced