RT-1: Robotics Transformer for Real-World Control at Scale

RT-1 (Robotics Transformer 1) is the seminal work that proved transformer models can control real robots at scale. Trained on 130K real-world robot demonstrations across 744 tasks (pick, place, open, close, insert, etc.) collected over 17 months in Google office kitchens, RT-1 is a 35M-parameter model that takes RGB images and natural language instructions as input and outputs discrete robot actions at 3 Hz. The architecture is simple but effective: FiLM-conditioned EfficientNet-B3 for visual encoding, tokenized actions, and a Transformer decoder. The key insight is generalization at scale: RT-1 can execute commands it was never explicitly trained on ("pick up the extinct animal" when shown a dinosaur toy) by leveraging the cross-task structure in the large training corpus. The model is remarkably robust to distractors, lighting changes, and background clutter—much more so than smaller IL models trained on single-task datasets. The authors also show that RT-1 can be distilled into smaller models (RT-1 Small, 6M parameters) that run at 10 Hz on edge hardware without major accuracy loss. RT-1's training pipeline is fully open-sourced: the dataset format (RLDS), the model architecture (TensorFlow + JAX), and the inference runtime (ROS2 integration). The codebase includes data loaders for BridgeData V2 and ALOHA, plus evaluation scripts for the SimplerEnv benchmark. While newer models (RT-2, Octo, OpenVLA) have surpassed RT-1 on absolute accuracy, it remains the gold standard for understanding how to build and scale real-world robot datasets. For practitioners, RT-1 is the starting point for training custom VLA policies: collect your own demonstrations in the RLDS format, fine-tune the pretrained RT-1 checkpoint, and deploy with the provided ROS2 node. The well-documented data pipeline and preprocessing scripts make it significantly easier than building from scratch.