RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Devesh K. Misra, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich · Google DeepMind, Google Research · 2023

Summary

RT-1 is a 35M-parameter transformer trained on 130K robot demonstrations that generalizes to new tasks, objects, and environments, forming the foundation for Google's RT-2 and RT-X line of VLA models.

Abstract Summary

RT-1 (Robotics Transformer 1) is the seminal work that proved transformer models can control real robots at scale. Trained on 130K real-world robot demonstrations across 744 tasks (pick, place, open, close, insert, etc.) collected over 17 months in Google office kitchens, RT-1 is a 35M-parameter model that takes RGB images and natural language instructions as input and outputs discrete robot actions at 3 Hz. The architecture is simple but effective: FiLM-conditioned EfficientNet-B3 for visual encoding, tokenized actions, and a Transformer decoder. The key insight is generalization at scale: RT-1 can execute commands it was never explicitly trained on ("pick up the extinct animal" when shown a dinosaur toy) by leveraging the cross-task structure in the large training corpus. The model is remarkably robust to distractors, lighting changes, and background clutter—much more so than smaller IL models trained on single-task datasets. The authors also show that RT-1 can be distilled into smaller models (RT-1 Small, 6M parameters) that run at 10 Hz on edge hardware without major accuracy loss. RT-1's training pipeline is fully open-sourced: the dataset format (RLDS), the model architecture (TensorFlow + JAX), and the inference runtime (ROS2 integration). The codebase includes data loaders for BridgeData V2 and ALOHA, plus evaluation scripts for the SimplerEnv benchmark. While newer models (RT-2, Octo, OpenVLA) have surpassed RT-1 on absolute accuracy, it remains the gold standard for understanding how to build and scale real-world robot datasets. For practitioners, RT-1 is the starting point for training custom VLA policies: collect your own demonstrations in the RLDS format, fine-tune the pretrained RT-1 checkpoint, and deploy with the provided ROS2 node. The well-documented data pipeline and preprocessing scripts make it significantly easier than building from scratch.

Key Points

35M-parameter transformer trained on 130K real robot demonstrations.
Generalizes to unseen tasks, objects, and environments.
Robust to distractors, lighting changes, background clutter.
Open-source: RLDS format, JAX model, ROS2 inference node.
Distillable to 6M parameters for edge deployment.

Preliminaries

We aim to learn robot policies to solve language-conditioned tasks from vision. Formally, we consider a sequential decision-making environment. At timestep $t=0$, the policy $\pi$ is presented with a language instruction $i$ and an initial image observation $x_{0}$. The policy produces an action distribution $\pi(\cdot~/~i,x_{0})$ from which an action $a_{0}$ is sampled and applied to the robot.

System Overview

The goal of this work is to build and demonstrate a general robot learning system that can absorb large amounts of data and generalize effectively. We use mobile manipulators from Everyday Robots 3 3 3 everydayrobots.com , which have a 7 degree-of-freedom arm, a two-fingered gripper, and a mobile base (see Fig. 2 (d)).

To collect data and evaluate our method, we use three kitchen-based environments: two real office kitchens and a training environment modelled off these real kitchens. The training environment, shown in Fig.

RT-1: Robotics Transformer

In this section, we describe how we tokenize the images, text, and actions, and then discuss the RT-1 model architecture. We then describe how we attain the runtime speed required for real-time control. Lastly, we describe the data collection procedure and the skills and instructions in our dataset.

Our model is built on a Transformer architecture ( Vaswani et al., 2017 ) and takes a history of images and task description as input and directly outputs tokenized actions, as shown in Fig. 1(a) and in detail in Fig. 3 .

RT-1: Robotics Transformer — RT-1: Robotics Transformer for Real-World Control at Scale — Figure 4: Evaluation scenarios for distractors (first row), from left to right: easy (0-5 distractors), medium (9 distractors), hard (9 distractors and occluded object); background (second row), from left to right: original environment, patterned table cloth, new kitchen; and realistic scenarios in the real kitchen (third row), generalization levels from left to right: $L1$, $L2$ and $L3$.

Experiments

Can an RT-1 learn to perform a large number of instructions, as well as to generalize in zero shot to new tasks, objects and environments? (Section 6.2 ) Can we push the resulting model even further by incorporating heterogeneous data sources, such as simulated data or data from different robots? (Section 6.3 ) How do various methods generalize to long-horizon robotic scenarios?

(Section 6.4 ) How do generalization metrics change with varying amounts of data quantity and data diversity? (Section 6.5 ) What are the important and practical decisi...

Experiments — RT-1: Robotics Transformer for Real-World Control at Scale — Table 2: Overall performance of RT-1 and baselines across seen tasks, generalization to unseen tasks, and robustness to distractors and backgrounds.

Sources and Demonstrations

Figures are reproduced from Brohan et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.