RT-2: Vision-Language-Action Models That Generalize to Novel Tasks

RT-2 (Robotics Transformer 2) is a watershed paper in vision-language-action modeling for robotics. Published by Google DeepMind at ICRA 2024, it takes a pre-trained vision-language model (PaLI-X or PaLM-E) and fine-tunes it to output robot action tokens — not text, but quantized continuous end-effector poses and gripper states. The resulting model is the first large-scale demonstration that general-purpose vision-language representations can be repurposed for closed-loop robot control. The key insight is co-training: the model is trained simultaneously on vision-language tasks (image captioning, VQA, web-scale text) and robotics demonstration data (camera images → action sequences). This preserves the language model's general world knowledge while teaching it to ground abstract semantic concepts to physical motor commands. When deployed, RT-2 can respond to novel linguistic instructions like "place the extinct animal near the tall landmark" even if the exact object configuration was never seen during robot training. RT-2 achieves significant improvements over RT-1 (its predecessor) and over specialist imitation-learning policies on generalization benchmarks. On real robot evaluation across kitchen tasks, it shows 2–3x improvement on novel object categories and unprecedented performance on long-horizon reasoning tasks. The model outputs action tokens autoregressively at 3 Hz, directly controlling a 7-DoF arm with a two-finger gripper. While the original training infrastructure is not publicly available (the model requires massive compute and proprietary data), the Open X-Embodiment dataset and the RT-X models released on Hugging Face provide a community path to replicate and extend the approach. RT-2 established the VLA paradigm that underpins virtually all modern generalist manipulation research, including OpenVLA, Octo, and π₀.