Robotics Research Hub
RT-1: Robotics Transformer for Real-World Control at Scale

RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich · Google DeepMind, Google Research, Everyday Robots · 2023

llm-robotics foundation-models imitation-learning
PDF arXiv Code (1723★)

Framework

TensorFlow

License

Apache-2.0

Stars

1,723

Summary

A large transformer model trained on 130k episodes of real robot manipulation to output discretized arm-and-gripper actions from RGB images and natural language instructions.

Abstract Summary

RT-1 (Robotics Transformer 1) is the foundational predecessor to RT-2 and marks one of the first successful applications of large-scale transformer models to real-world robot control. Developed by a large team at Google DeepMind, Google Research, and Everyday Robots, RT-1 is trained on a dataset of over 130,000 real-world episodes collected across 13 months using a fleet of mobile manipulators in office kitchen environments. It demonstrated that high-capacity sequence models, when fed with enough diverse robotic data, could generalize across tasks, robots, and environmental settings far better than previous specialist policies. The model architecture is a decoder-only transformer that ingests a history of RGB images and a natural language instruction, then predicts discretized motor commands for a 7-DOF arm, a base, and a gripper. Importantly, RT-1 does not use privileged state information; it learns entirely from pixels and text. Action discretization converts continuous joint positions into tokens, allowing the model to be trained with standard next-token cross-entropy loss. This unification of vision, language, and action within a single autoregressive framework proved remarkably effective, achieving over 97% success rate on a broad set of manipulation skills in previously unseen environments. One of the most significant contributions of RT-1 is its evidence for the scaling law of robotics data. By training on orders of magnitude more real robot interactions than prior work, the authors observe clear performance improvements that mirror the trends seen in language and vision models. The model also shows positive transfer when trained on datasets from multiple robots, suggesting that a shared representation of manipulation is learnable across embodiments. For the research community, this paper validated the hypothesis that robotic foundation models could follow the same data-scaling playbook as NLP and vision. RT-1 also serves as the architectural backbone for the Open X-Embodiment dataset and Octo. Many subsequent papers cite RT-1 as proof that a unified action-generation model, rather than a modular perception-planning-control stack, can solve complex long-horizon tasks when grounded in large-scale data. Its success directly paved the way for RT-2 by showing that the transformer paradigm could accommodate both visual and action modalities without specialized robotics-specific inductive biases beyond tokenization.

Key Points

  • First large-scale transformer model trained end-to-end on real robot manipulation.
  • 130K+ episodes across 13 months on Everyday Robots mobile manipulators.
  • Pixel-to-action control with natural language instructions; no privileged state.
  • Demonstrated scaling benefits and positive cross-robot transfer.
  • Precursor to RT-2, Open X-Embodiment, and the broader VLA research wave.

Preliminaries

We aim to learn robot policies to solve language-conditioned tasks from vision. Formally, we consider a sequential decision-making environment. At timestep $t=0$, the policy $\pi$ is presented with a language instruction $i$ and an initial image observation $x_{0}$. The policy produces an action distribution $\pi(\cdot~/~i,x_{0})$ from which an action $a_{0}$ is sampled and applied to the robot.

Preliminaries — RT-1: Robotics Transformer for Real-World Control at Scale
Figure 2: (a) Robot classroom where we collect data at scale; (b) a real office kitchen, one of the two realistic environments used for evaluation (named Kitchen1 in the rest of the paper); (c) a different office kitchen used for evaluation (named Kitchen2 in the rest of the paper); (d) mobile manipulator used throughout the paper; (e) a set of objects used for most of the skills to expand skill diversity; (f) a more diverse set of objects used mostly to expand object diversity of the picking skill.

System Overview

The goal of this work is to build and demonstrate a general robot learning system that can absorb large amounts of data and generalize effectively. We use mobile manipulators from Everyday Robots 3 3 3 everydayrobots.com , which have a 7 degree-of-freedom arm, a two-fingered gripper, and a mobile base (see Fig. 2 (d)).

To collect data and evaluate our method, we use three kitchen-based environments: two real office kitchens and a training environment modelled off these real kitchens. The training environment, shown in Fig.

System Overview — RT-1: Robotics Transformer for Real-World Control at Scale
Figure 3: The architecture diagram of RT-1. The instruction is transformed into a USE embedding and used to condition a pre-trained EfficientNet via FiLM layers. The resulting vision-language tokens are reduced by the TokenLearner and fed into a decoder-only Transformer, which outputs tokenized actions.

RT-1: Robotics Transformer

In this section, we describe how we tokenize the images, text, and actions, and then discuss the RT-1 model architecture. We then describe how we attain the runtime speed required for real-time control. Lastly, we describe the data collection procedure and the skills and instructions in our dataset.

Our model is built on a Transformer architecture ( Vaswani et al., 2017 ) and takes a history of images and task description as input and directly outputs tokenized actions, as shown in Fig. 1(a) and in detail in Fig. 3 .

RT-1: Robotics Transformer — RT-1: Robotics Transformer for Real-World Control at Scale
Figure 4: Evaluation scenarios for distractors (first row), from left to right: easy (0-5 distractors), medium (9 distractors), hard (9 distractors and occluded object); background (second row), from left to right: original environment, patterned table cloth, new kitchen; and realistic scenarios in the real kitchen (third row), generalization levels from left to right: $L1$, $L2$ and $L3$.

Experiments

Can an RT-1 learn to perform a large number of instructions, as well as to generalize in zero shot to new tasks, objects and environments? (Section 6.2 ) Can we push the resulting model even further by incorporating heterogeneous data sources, such as simulated data or data from different robots? (Section 6.3 ) How do various methods generalize to long-horizon robotic scenarios?

(Section 6.4 ) How do generalization metrics change with varying amounts of data quantity and data diversity? (Section 6.5 ) What are the important and practical decisi...

Experiments — RT-1: Robotics Transformer for Real-World Control at Scale
Table 2: Overall performance of RT-1 and baselines across seen tasks, generalization to unseen tasks, and robustness to distractors and backgrounds.

Sources and Demonstrations

Figures are reproduced from Brohan et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.

Share

Related Papers