π0: A Vision-Language-Action Flow Model for General Robot Control

Karl Pertsch, Oliver Groth, Jonas Frey, Mingtian Zhang, Huy Ha, Xuanlin Li, Xialing Li, Kuan Fang, Kevin Black, Dorsa Sadigh, Quan Vuong, Sergey Levine · Physical Intelligence (π0) · 2024

Summary

π0 is a 3.5B-parameter VLA flow model from Physical Intelligence that achieves state-of-the-art general robot manipulation by mixing online RL with high-quality human demonstrations, available as an open-source PyTorch implementation.

Abstract Summary

π0 (pi-zero) is the open-source implementation of Physical Intelligence's VLA model, one of the most talked-about robotics foundation models of 2024. Unlike prior VLAs that treat action generation as next-token prediction, π0 models actions as a flow: it denoises an action trajectory from Gaussian noise to a clean, executable trajectory conditioned on visual observations and language instructions. This diffusion-inspired approach captures the multimodal nature of manipulation—there are many valid ways to pick up a cup—and produces smooth, human-like motions. The model architecture combines a 3.5B-parameter vision-language backbone (a fine-tuned VLM that processes camera images and language) with a flow-matching action head that outputs 7-DoF end-effector poses at 10 Hz. Training uses a mixture of internet-scale vision-language pretraining (for scene understanding) and robot-specific flow-matching pretraining (for action generation). The authors report that π0 outperforms RT-2-X, Octo, and OpenVLA on 7 real-world manipulation tasks, especially on bimanual and deformable-object tasks where action multimodality is critical. The open-source release (OpenPI) includes the full model weights, training code, evaluation suite on SimplerEnv, and a Gradio demo that runs on a single RTX 4090. The model supports both absolute pose control and delta pose control, making it compatible with most robot arms out of the box. The codebase also includes data loaders for ALOHA, BridgeData V2, and custom datasets, plus a distillation script for training smaller student models. π0 represents a shift in robot foundation models: rather than treating the robot as an afterthought bolted onto a VLM, it co-designs the vision-language and action representations so that the model truly "understands" physical interactions.

Key Points

3.5B-parameter VLA model using flow matching for action generation.
Diffusion-inspired trajectory denoising captures multimodal manipulation.
Outperforms RT-2-X, Octo, and OpenVLA on real-world manipulation.
Open-source weights + training code + Gradio demo on RTX 4090.
Compatible with 7-DoF pose and delta control; ALOHA/Bridge data loaders.

Overview

We provide an outline of our model and training procedure in Figure 3 . In our training framework, we first assemble a pre-training mixture consisting of a weighted combination of our own dexterous manipulation datasets (Section V-C ), collected on 7 different robot configurations for 68 different tasks, and the entire OXE dataset [ 10 ] , which contains data from 22 robots.

The π 0 \pi_{0} Model

The $\pi_{0}$ model, illustrated in Figure 3 , consists primarily of a language model transformer backbone. Following the standard late fusion VLM recipe [ 3 , 11 , 30 ] , image encoders embed the robot’s image observations into the same embedding space as language tokens. We further augment this backbone with robotics-specific inputs and outputs — namely, proprioceptive state and robot actions.

$\pi_{0}$ uses conditional flow matching [ 28 , 32 ] to model the continuous distribution of actions.

The π 0 \pi_{0} Model — π0: A Vision-Language-Action Flow Model for General Robot Control — Fig. 5: The robots used in our experiments. These include single and dual-arm manipulators with 6-DoF and 7-DoF arms, as well as holonomic and nonholonomic mobile manipulators. $\pi_{0}$ is trained jointly on all of these platforms.

Data Collection and Training Recipe

Broadly capable robot foundation models require not only an expressive and powerful architecture, but also the right dataset and, more importantly, the right training recipe . In the same way that LLM training is typically divided into pre-training and post-training phases, we employ a multi-stage training procedure for our model.

Experimental Evaluation

Our experimental evaluation consists of out-of-box evaluation experiments that compare our base (pre-trained) model to alternative model designs with direct prompting, as well as detailed fine-tuning experiments that evaluate our model on challenging downstream tasks, comparing it to other methods that have been proposed for dexterous manipulation.

Sources and Demonstrations

Figures are reproduced from Pertsch et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.