RT-1: Robotics Transformer for Real-World Control at Scale

RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich · Google DeepMind, Google Research, Everyday Robots · 2023

Framework

TensorFlow

License

Apache-2.0

Stars

1,723

Summary

A large transformer model trained on 130k episodes of real robot manipulation to output discretized arm-and-gripper actions from RGB images and natural language instructions.

Abstract Summary

RT-1 (Robotics Transformer 1) is the foundational predecessor to RT-2 and marks one of the first successful applications of large-scale transformer models to real-world robot control. Developed by a large team at Google DeepMind, Google Research, and Everyday Robots, RT-1 is trained on a dataset of over 130,000 real-world episodes collected across 13 months using a fleet of mobile manipulators in office kitchen environments. It demonstrated that high-capacity sequence models, when fed with enough diverse robotic data, could generalize across tasks, robots, and environmental settings far better than previous specialist policies. The model architecture is a decoder-only transformer that ingests a history of RGB images and a natural language instruction, then predicts discretized motor commands for a 7-DOF arm, a base, and a gripper. Importantly, RT-1 does not use privileged state information; it learns entirely from pixels and text. Action discretization converts continuous joint positions into tokens, allowing the model to be trained with standard next-token cross-entropy loss. This unification of vision, language, and action within a single autoregressive framework proved remarkably effective, achieving over 97% success rate on a broad set of manipulation skills in previously unseen environments. One of the most significant contributions of RT-1 is its evidence for the scaling law of robotics data. By training on orders of magnitude more real robot interactions than prior work, the authors observe clear performance improvements that mirror the trends seen in language and vision models. The model also shows positive transfer when trained on datasets from multiple robots, suggesting that a shared representation of manipulation is learnable across embodiments. For the research community, this paper validated the hypothesis that robotic foundation models could follow the same data-scaling playbook as NLP and vision. RT-1 also serves as the architectural backbone for the Open X-Embodiment dataset and Octo. Many subsequent papers cite RT-1 as proof that a unified action-generation model, rather than a modular perception-planning-control stack, can solve complex long-horizon tasks when grounded in large-scale data. Its success directly paved the way for RT-2 by showing that the transformer paradigm could accommodate both visual and action modalities without specialized robotics-specific inductive biases beyond tokenization.

Key Points

  • First large-scale transformer model trained end-to-end on real robot manipulation.
  • 130K+ episodes across 13 months on Everyday Robots mobile manipulators.
  • Pixel-to-action control with natural language instructions; no privileged state.
  • Demonstrated scaling benefits and positive cross-robot transfer.
  • Precursor to RT-2, Open X-Embodiment, and the broader VLA research wave.

Additional Notes

Training Tips

  • Data diversity matters more than data volume alone; include many objects, lighting conditions, and backgrounds.
  • Action token vocabulary size impacts coarse-vs-fine control precision.
  • The model is sensitive to image preprocessing; matching the training resolution and normalization is critical.
  • RT-2 (Brohan et al., 2023)
  • Open X-Embodiment (Padalkar et al., 2023)
  • Octo (Team et al., 2024)
Share

Related Papers

X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

Kai Xiong, Hongjie Fang, Lixin Yang et al. · arXiv · May 2026

Effectively handling the interplay between spatial perception and action generation remains a critical bottleneck in robotic manipulation. Existing methods typically treat spatial perception and action execution as decoupled or strictly...

imitation-learning manipulation foundation-models
PDF Intermediate
No code repo May 2026