Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration · Google DeepMind and 20+ institutions worldwide · 2023

Summary

The largest collaborative robot learning dataset ever assembled, spanning 22 robots from 21 institutions, enabling training of generalist RT-X policies that exhibit positive cross-embodiment transfer.

Abstract Summary

Open X-Embodiment represents an unprecedented community-scale effort to consolidate the fragmented landscape of robot learning datasets. The collaboration spans 21 institutions and 22 distinct robotic platforms—from single-arm tabletop manipulators to mobile manipulators and humanoids—resulting in a unified dataset of over 160,000 real-world manipulation tasks demonstrating 527 skills. Each trajectory was meticulously standardized into a common data format with consistent action spaces, observation modalities, and metadata schemas, making it feasible to train a single policy across hardware that was previously considered incompatible. This cross-embodiment dataset naturally raises the question: can a single model learn from such heterogeneous data and transfer knowledge across platforms? The authors provide a definitive affirmative answer in the form of RT-X, a high-capacity transformer-based policy trained end-to-end on the full Open X-Embodiment dataset. RT-X demonstrates strong positive transfer: robots trained with the generalist model outperform their specialist counterparts when evaluated on held-out tasks. A Franka robot, for instance, achieves higher success rates after being initialized with RT-X than after training only on Franka-specific data. This suggests that the generalist model extracts shared physical priors—such as how to grasp, push, or assemble—that transcend individual morphologies. The dataset is released alongside a set of baseline models and rigorous evaluation protocols. The authors systematically investigate which training strategies work best for multi-robot pre-training, finding that action tokenization and cross-embodiment batching are critical for stable optimization. They also show that data diversity—spanning different grippers, cameras, and environments—is more important than raw quantity for generalization. The standardized data format and open release are expected to serve as the bedrock for the next generation of generalist robot policies, much as ImageNet did for computer vision and the Pile did for language modeling. Long-term, Open X-Embodiment aims to catalyze the same consolidation seen in NLP and computer vision: from bespoke per-robot solutions to unified foundation models that can be fine-tuned with modest data for any new platform. The project website provides dataset viewers, download tools, and integration guides, dramatically lowering the entry barrier for labs that wish to contribute to or leverage this shared resource.

Key Points

Largest open robot learning dataset: 160k+ tasks and 527 skills across 22 robots and 21 institutions.
Standardized data format enabling training of single policies on heterogeneous robot hardware.
RT-X generalist policy shows positive transfer and outperforms specialist policies on held-out tasks.
Action tokenization and cross-embodiment batching are key to multi-robot pre-training stability.
Open-sourced dataset, baselines, and evaluation protocols to accelerate community research.

The Open X-Embodiment Repository

We introduce the Open X-Embodiment Repository ( robotics-transformer-x.github.io ) – an open-source repository which includes large-scale data along with pre-trained model checkpoints for X-embodied robot learning research.

More specifically, we provide and maintain the following open-source resources to the broader community: Open X-Embodiment Dataset : robot learning dataset with 1M+ robot trajectories from $22$ robot embodiments . Pre-Trained Checkpoints : a selection of RT-X model checkpoints ready for inference and finetuning.

The Open X-Embodiment Repository — Open X-Embodiment: Robotic Learning Datasets and RT-X Models — Fig. 3 : RT-1-X mean success rate is $50\%$ higher than that of either the Original Method or RT-1. RT-1 and RT-1-X have the same network architecture. Therefore the performance increase can be attributed to co-training on the robotics data mixture. The lab logos indicate the physical location of real robot evaluation, and the robot pictures indicate the embodiment used for the evaluation.

RT-X Design

To evaluate how much X-embodiment training can improve the performance of learned policies on individual robots, we require models that have sufficient capacity to productively make use of such large and heterogeneous datasets. To that end, our experiments will build on two recently proposed Transformer-based robotic policies: RT-1 [ 8 ] and RT-2 [ 9 ] .

We briefly summarize the design of these models in this section, and discuss how we adapted them to the X-embodiment setting in our experiments.

Experimental Results

Our experiments answer three questions about the effect of X-embodiment training: (1) Can policies trained on our X-embodiment dataset effectively enable positive transfer, such that co-training on data collected on multiple robots improves performance on the training task? (2) Does co-training models on data from multiple platforms and tasks improve generalization to new, unseen tasks?

(3) What is the influence of different design dimensions, such as model size, model architecture or dataset composition, on performance and generalization capab...

Sources and Demonstrations

Figures are reproduced from Collaboration See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.