Octo: An Open-Source Generalist Robot Policy

Octo: An Open-Source Generalist Robot Policy

Featured

Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, Sergey Levine · UC Berkeley, Stanford, TRI · 2024

Framework

JAX / Flax

License

MIT

Stars

1,652

Summary

Octo is a large open-source transformer-based generalist robot policy trained on 800k trajectories, supporting language-conditioned and goal-image-conditioned control across 9 robotic platforms with efficient fine-tuning on consumer GPUs.

Abstract Summary

Octo addresses a critical gap in robot foundation models: while proprietary generalist policies like RT-2 showed impressive results, there was no open-source alternative that researchers could freely fine-tune, inspect, and modify. Octo fills this niche. Trained on 800,000 trajectories from the Open X-Embodiment dataset, Octo is a transformer-based policy capable of conditioning on language commands or goal images and generating actions for diverse sensor and actuator configurations. It supports both visual and proprioceptive observations and can output continuous robot actions at interactive rates. The architecture is designed for practical flexibility. Unlike prior work that couples the policy tightly to specific robot hardware, Octo employs an observation-action tokenizer that projects heterogeneous sensor streams into a common token space. This means the same pretrained backbone can be fine-tuned to a new robot with different camera counts, image resolutions, or action dimensions in just a few hours on a single consumer GPU. The authors demonstrate this cross-platform capability on 9 robotic platforms, ranging from low-cost WidowX arms to Franka Research robots and mobile manipulators, confirming that Octo serves as an effective policy initialization across the board. Training at this scale required careful engineering. The authors release their full pre-training pipeline, including data loading, tokenization, and distributed training scripts, all built on top of JAX and Flax. They conduct extensive ablations on architectural choices—such as whether to condition on language tokens early or late in the transformer stack—and on dataset curation, finding that data diversity and quality filtering matter more than sheer scale for generalization. These ablation studies provide valuable guidance for future efforts in building generalist robot models. Empirically, Octo achieves competitive or superior performance compared to prior open methods and serves as a strong initialization for downstream tasks. When fine-tuned on a small amount of in-domain data, it quickly adapts to new manipulation skills, object geometries, and environments that were not present in the pre-training dataset. The authors also provide a Hugging Face integration, making it possible to download pretrained weights and fine-tune models with a few lines of Python. For the robot learning community, Octo represents a foundational open generalist policy that can be used as a starting point for a wide variety of research projects.

Key Points

  • Large open-source generalist policy trained on 800k trajectories from Open X-Embodiment.
  • Supports language and goal-image conditioning with a flexible transformer architecture.
  • Observation-action tokenizer enables cross-robot fine-tuning on consumer GPUs within hours.
  • Evaluated on 9 distinct robot platforms with strong few-shot transfer to new tasks.
  • Full training and fine-tuning stack released in JAX/Flax with Hugging Face integration.

Additional Notes

Fine-tuning Tips

  • Use the provided observation tokenizer config to map your robot’s cameras to the pretrained input space.
  • Fine-tuning works best with 50–500 in-domain trajectories; less data often leads to overfitting.
  • The JAX ecosystem benefits from TPUs for large-scale pre-training, but GPU fine-tuning is well supported.
  • Open X-Embodiment (Padalkar et al., 2023)
  • RT-2 (Brohan et al., 2023)
  • OpenVLA (Kim et al., 2024)
Share

Related Papers

X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

X-Imitator: Spatial-Aware Imitation Learning via Bidirectional Action-Pose Interaction

Kai Xiong, Hongjie Fang, Lixin Yang et al. · arXiv · May 2026

Effectively handling the interplay between spatial perception and action generation remains a critical bottleneck in robotic manipulation. Existing methods typically treat spatial perception and action execution as decoupled or strictly...

imitation-learning manipulation foundation-models
PDF Intermediate
No code repo May 2026
DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset

DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset

Alexander Khazatsky, Karl Pertsch, Suraj Nair et al. · RSS 2024 · Jul 2024

A 350-hour dataset of diverse real-world robot manipulation across 22 robots in 71 scenes, designed to train scalable and generalist imitation learning policies.

imitation-learning manipulation foundation-models
Code PDF Advanced
GitHub ★ 362 Code updated: May 2026
LeRobot: A Library for Real-World Robot Learning

LeRobot: A Library for Real-World Robot Learning

Remi Cadene, Simon Alibert, Alexander Soare et al. · NeurIPS 2024 Workshop · Jun 2024

Hugging Face's LeRobot is an open-source PyTorch framework providing pretrained models, datasets, and training scripts for imitation and reinforcement learning on real robots, lowering the entry barrier to robot learning.

imitation-learning rl foundation-models manipulation
Code PDF Beginner
GitHub ★ 24,333 Code updated: May 2026
Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration · arXiv · Oct 2023

The largest collaborative robot learning dataset ever assembled, spanning 22 robots from 21 institutions, enabling training of generalist RT-X policies that exhibit positive cross-embodiment transfer.

imitation-learning foundation-models manipulation dataset
Code PDF Intermediate
GitHub ★ 1,853 Code updated: May 2026