Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Featured

Open X-Embodiment Collaboration · Google DeepMind and 20+ institutions worldwide · 2023

Framework

Jupyter Notebook

License

Apache-2.0

Stars

1,853

Summary

The largest collaborative robot learning dataset ever assembled, spanning 22 robots from 21 institutions, enabling training of generalist RT-X policies that exhibit positive cross-embodiment transfer.

Abstract Summary

Open X-Embodiment represents an unprecedented community-scale effort to consolidate the fragmented landscape of robot learning datasets. The collaboration spans 21 institutions and 22 distinct robotic platforms—from single-arm tabletop manipulators to mobile manipulators and humanoids—resulting in a unified dataset of over 160,000 real-world manipulation tasks demonstrating 527 skills. Each trajectory was meticulously standardized into a common data format with consistent action spaces, observation modalities, and metadata schemas, making it feasible to train a single policy across hardware that was previously considered incompatible. This cross-embodiment dataset naturally raises the question: can a single model learn from such heterogeneous data and transfer knowledge across platforms? The authors provide a definitive affirmative answer in the form of RT-X, a high-capacity transformer-based policy trained end-to-end on the full Open X-Embodiment dataset. RT-X demonstrates strong positive transfer: robots trained with the generalist model outperform their specialist counterparts when evaluated on held-out tasks. A Franka robot, for instance, achieves higher success rates after being initialized with RT-X than after training only on Franka-specific data. This suggests that the generalist model extracts shared physical priors—such as how to grasp, push, or assemble—that transcend individual morphologies. The dataset is released alongside a set of baseline models and rigorous evaluation protocols. The authors systematically investigate which training strategies work best for multi-robot pre-training, finding that action tokenization and cross-embodiment batching are critical for stable optimization. They also show that data diversity—spanning different grippers, cameras, and environments—is more important than raw quantity for generalization. The standardized data format and open release are expected to serve as the bedrock for the next generation of generalist robot policies, much as ImageNet did for computer vision and the Pile did for language modeling. Long-term, Open X-Embodiment aims to catalyze the same consolidation seen in NLP and computer vision: from bespoke per-robot solutions to unified foundation models that can be fine-tuned with modest data for any new platform. The project website provides dataset viewers, download tools, and integration guides, dramatically lowering the entry barrier for labs that wish to contribute to or leverage this shared resource.

Key Points

  • Largest open robot learning dataset: 160k+ tasks and 527 skills across 22 robots and 21 institutions.
  • Standardized data format enabling training of single policies on heterogeneous robot hardware.
  • RT-X generalist policy shows positive transfer and outperforms specialist policies on held-out tasks.
  • Action tokenization and cross-embodiment batching are key to multi-robot pre-training stability.
  • Open-sourced dataset, baselines, and evaluation protocols to accelerate community research.

Additional Notes

Getting Started

  • Clone the dataset repository and use the provided TFDS pipeline for efficient loading.
  • For training RT-X from scratch, the authors recommend starting with pre-computed action tokenization.
  • Evaluation scripts assume standard camera intrinsics; recalibrate if using custom hardware.
  • RT-1 (Brohan et al., 2022)
  • RT-2 (Brohan et al., 2023)
  • Octo (Ghosh et al., 2024)
Share

Related Papers

MonoDuo: Using One Robot Arm to Learn Bimanual Policies

MonoDuo: Using One Robot Arm to Learn Bimanual Policies

Sandeep Bajamahal, Lawrence Yunliang Chen, Toru Lin et al. · ICRA 2026 · May 2026

Bimanual coordination is essential for many real-world manipulation tasks, yet learning bimanual robot policies is limited by the scarcity of bimanual robots and datasets. Single-arm robots, however,

manipulation bimanual imitation-learning dataset
PDF Intermediate
No code repo Code updated: May 2026
X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

Kai Xiong, Hongjie Fang, Lixin Yang et al. · arXiv · May 2026

Effectively handling the interplay between spatial perception and action generation remains a critical bottleneck in robotic manipulation. Existing methods typically treat spatial perception and action execution as decoupled or strictly...

imitation-learning manipulation foundation-models
PDF Intermediate
No code repo May 2026
DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset

DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset

Alexander Khazatsky, Karl Pertsch, Suraj Nair et al. · RSS 2024 · Jul 2024

A 350-hour dataset of diverse real-world robot manipulation across 22 robots in 71 scenes, designed to train scalable and generalist imitation learning policies.

imitation-learning manipulation foundation-models
Code PDF Advanced
GitHub ★ 362 Code updated: May 2026
LeRobot: A Library for Real-World Robot Learning

LeRobot: A Library for Real-World Robot Learning

Remi Cadene, Simon Alibert, Alexander Soare et al. · NeurIPS 2024 Workshop · Jun 2024

Hugging Face's LeRobot is an open-source PyTorch framework providing pretrained models, datasets, and training scripts for imitation and reinforcement learning on real robots, lowering the entry barrier to robot learning.

imitation-learning rl foundation-models manipulation
Code PDF Beginner
GitHub ★ 24,333 Code updated: May 2026